NanoMQ 0.21.1 is a republish of 0.21.0 in order to bring some important fixes such as : https://github.com/nanomq/nanomq/issues/1632
This Major release has two big new features: MQTT Stream and a New Plugin system along with multiple important fixes and minor new features.
- Read bridging backoff time from the conf file.
- Add rpath support, allow users to specify their own so file to link
- Fix Bridging aio overflow while using HTTP Publish API with bridging
- Add a dependency checker for DDS Proxy to ensure both sides using the same ver of CycloneDDS
- Fix a stack overflow bug caused by compiler optimization
- Expose TCP level option for MQTT TCP bridging
- Add a new Plugin system. It allows the user to set a self-defined user property.
- Add a new conf option to disable/enable MQTT TCP listener
- Fix a potential decoding bug of a large CONNECT Packet
- Sync NNG with Upstream to fix the duplicated ClientID issue.
- Add a brand new Messaging Queue feature: MQTT Stream, which automatically partitions MQTT msgs to disk. Allows consuming msg via NNG Req/Rep Sock.
MQTT STREAM
A basic version of MQ : MQTT STREAM has been released, and its main ability is to persist the content of mqtt messages, for consumer to retrieve msgs by key.
Configuration
First we introduce three data structures for MQTT STREAM.
- exchange_server/exchange/ringbus
The specific reference configuration is as follows.
exchange_client.mq1 {
# # exchanges contains multiple MQ exchanger
exchange {
# # MQTT Topic for filtering messages and saving to queue
topic = "exchange/topic1",
# # MQ name
name = "exchange_no1",
# # MQ category. Only support Ringbus for now
ringbus = {
# # ring buffer name
name = "ringbus",
# # max length of ring buffer (msg count)
cap = 1000
fullOp = 3
}
}
}
Notice:
- The topic indicates which topic you want to put data in ringbus.
- Where fullOp is a key attribute and represents the operation when the ringbus is full.
- 0: RB_FULL_NONE: When the ringbus is full, no action is taken and the message enqueue fail
- 1: RB_FULL_DROP: When the ringbus is full, the data in the ringbus is discarded
- 2: RB_FULL_RETURN: When the ringbus is full, the data in the ringbus is taken out and returned to the aio
- 3: RB_FULL_FILE: When the ringbus is full, the data in the ringbus is written to the file
For example, when the ringbus is full and you want to store the data to a file, you should use fullOp=3
Messages in ringbus
Messages in ringbus are stored in the form of key/value, and currently the key is calculated using hash(clientid+topic+id).
How to query messages
If you want to query messages stored in a file, there's an interface, as well as a consumer demo.
First compile exchange_consumer. You can find the source code at
./nng/demo/exchange_consumer/exchange_consumer.c
Use exchange_consumer
- dump the msg data of all files.
./demo/exchange_consumer/exchange_consumer "dumpfile"
- dump the msg data from the file according to the key.
./demo/exchange_consumer/exchange_consumer "dumpkey:4862831023852633869"
- dump the msg data from the file according to the keys.
./demo/exchange_consumer/exchange_consumer "dumpkeys:2062146488009373518,2625100014974548622"
What's Changed in NanoMQ
- chore(ci): modify release api payload by @Swilder-M in https://github.com/nanomq/nanomq/pull/1539
- Set bridge backoff from configuration file by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1540
- Several fix by @OdyWayne in https://github.com/nanomq/nanomq/pull/1545
- add switch for static link by @OdyWayne in https://github.com/nanomq/nanomq/pull/1548
- Add rpath support. by @lee-emqx in https://github.com/nanomq/nanomq/pull/1554
- Add rpath support(cli). by @lee-emqx in https://github.com/nanomq/nanomq/pull/1555
- A basic Ringbus MQ is introduced. Not feasible feature still. by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1552
- FIX [nng] move nng head again for https://github.com/nanomq/nanomq/… by @JaylinYu in https://github.com/nanomq/nanomq/pull/1557
- CI fix by @OdyWayne in https://github.com/nanomq/nanomq/pull/1558
- Support send msgs to exchange and flush msgs from exchange to parquet by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1560
- Support read config file for parquet. by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1562
- Fix the memory leakage in smsgs. by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1565
- Fix #1568 by @JaylinYu in https://github.com/nanomq/nanomq/pull/1569
- FIX [workflow] remove mysql in build package by @OdyWayne in https://github.com/nanomq/nanomq/pull/1567
- Cancel flush when the flush aio is busy & Fix some memory leakage. by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1566
- SYNC [nng] move nng head by @OdyWayne in https://github.com/nanomq/nanomq/pull/1571
- Pass key to exchange instead of address & Add log to show the reason why flushing failed by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1570
- FIX [hook] fix sub crash by @JaylinYu in https://github.com/nanomq/nanomq/pull/1574
- Minor fix + new config by @JaylinYu in https://github.com/nanomq/nanomq/pull/1576
- Support handle msgs from external in HOOK_WAIT state. by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1572
- FIX [hook] Fix the wrong aio result & Update the topic. by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1578
- Support search msgs both from exchange and parquet by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1579
- NEW [hook] Use timestamp as the key to enqueue. by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1580
- Typo acl.md by @dferrarigrowtech in https://github.com/nanomq/nanomq/pull/1583
- Add parquet && exchange docs. by @lee-emqx in https://github.com/nanomq/nanomq/pull/1584
- MDF [broker] modify state machine to enable MQ while disabling webhook by @JaylinYu in https://github.com/nanomq/nanomq/pull/1586
- Support parsing new json format from external hook by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1585
- Support start MQ independently. by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1587
- Fix the error that some msgs from external are forwarded to webhook handler by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1588
- enable overide retain flag in bridge by @OdyWayne in https://github.com/nanomq/nanomq/pull/1589
- serveral fixes by @OdyWayne in https://github.com/nanomq/nanomq/pull/1592
- Support handle cmd in json message which from external hook by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1593
- Support fuzzing search on exchange and parquet (Old key and offset searching mode is obsolete) by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1598
- Support new exchange structure by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1595
- [WIP] * NEW [broker] add listener for exchange & MQ by @JaylinYu in https://github.com/nanomq/nanomq/pull/1597
- Fix a wrong entry of hook_cb in END state by @JaylinYu in https://github.com/nanomq/nanomq/pull/1601
- MDF [test] mdf topics for bridge test to avoid misuse by @OdyWayne in https://github.com/nanomq/nanomq/pull/1602
- Add commented codes that insert increment key to MQ. by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1604
- Fix the error that choosing wrong parser for a dds msg. & Fix memory leakage when dds take failed by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1605
- Plugin system is ready by @RanMaoyi in https://github.com/nanomq/nanomq/pull/1606
- Sync parquet_find_span modify. by @lee-emqx in https://github.com/nanomq/nanomq/pull/1608
- Add dds version check. by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1612
- Fixed nanomq stop cmd does not work issue. by @lee-emqx in https://github.com/nanomq/nanomq/pull/1613
- Update docs. by @lee-emqx in https://github.com/nanomq/nanomq/pull/1614
- FIx # 1616 by @JaylinYu in https://github.com/nanomq/nanomq/pull/1617
- FIX [broker] restore ASAN signal handler by @JaylinYu in https://github.com/nanomq/nanomq/pull/1621
- Fixed update bridge configuration example error && add some comments. by @lee-emqx in https://github.com/nanomq/nanomq/pull/1620
- enable more tcp option for bridge by @OdyWayne in https://github.com/nanomq/nanomq/pull/1615
- Add plugins docs and update docs for ringbus full operation by @RanMaoyi in https://github.com/nanomq/nanomq/pull/1623
- MDF [nng] move head for ci and new fix release by @JaylinYu in https://github.com/nanomq/nanomq/pull/1627
- FIX [broker] Fix the num_ctx in hook_init. by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1628
- Sync parquet write modify. by @lee-emqx in https://github.com/nanomq/nanomq/pull/1630
- Update the key to exchange. key=hash(clientid+topic+id). by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1609
- FIX [hook] Fix the error in building. by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1635
- MDF [nng] move nng head for next release by @JaylinYu in https://github.com/nanomq/nanomq/pull/1636
- Update Docs by @wanghaEMQ in https://github.com/nanomq/nanomq/pull/1637
- MDF [nng] move NNG head (new fix to include) by @JaylinYu in https://github.com/nanomq/nanomq/pull/1638
New Contributors
- @dferrarigrowtech made their first contribution in https://github.com/nanomq/nanomq/pull/1583
Full Changelog: https://github.com/nanomq/nanomq/compare/0.20.8...0.21
What's Changed in NanoNNG
- Introduce LinkedList in nanolib by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/736
- Fix the error in building for darwin arch. by @wanghaEMQ in https://github.com/nanomq/NanoNNG/pull/741
- MDF [mqtt/transport] reduce backoff starting time to 0-1s by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/739
- Fix the error in properties setting in connack. by @wanghaEMQ in https://github.com/nanomq/NanoNNG/pull/740
- Support read backoff max from configuration. by @wanghaEMQ in https://github.com/nanomq/NanoNNG/pull/743
- Linkedlist ut by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/742
- ringbuffer ut and rule engine support by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/744
- Support producer & exchange & exchange_client by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/745
- [MQ]Bind ring buffer with exchange & Store nng_msg in ringbuffer etc. by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/746
- change static link for mbedtls & quic by @OdyWayne in https://github.com/nanomq/NanoNNG/pull/748
- Support get ring buffer with name from exchange_client by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/747
- ADD [cmake] add switch for static link by @OdyWayne in https://github.com/nanomq/NanoNNG/pull/749
- ADD [cmake] add switch for static link fix by @OdyWayne in https://github.com/nanomq/NanoNNG/pull/750
- [WIP] Ringbuffer with key and export some api for get messages from exchange_client by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/752
- Resolve header dependence issues && UT of exchange_client get msgs apis by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/753
- tlstran_pipe_stop should free cparam by @lkyjason in https://github.com/nanomq/NanoNNG/pull/755
- MDF [cmake/NNGOptions] disable UT when build with static libs by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/756
- fix exchange by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/757
- Sync with upstream nng by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/758
- Fix wrong type def by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/759
- FIX [exchange] uint is not defined by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/760
- One exchange_client has one exchange and one exchange own one ringbuffer && void *** param for get msgs return by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/761
- broker_tcp/tls sync by @OdyWayne in https://github.com/nanomq/NanoNNG/pull/762
- When ringbuffer is full, get all msgs to user and clean up the ringbuffer by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/764
- MDF [transport/mqtt] code formalization + use MACRO by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/766
- Add aios for sending msgs to exchange and parquet by @wanghaEMQ in https://github.com/nanomq/NanoNNG/pull/765
- Fix ringbuffer exchange exchange_client UT by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/767
- First version of parquet write.(still need polish) by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/763
- Use lmq to store send messages and finish aio with error when error occurred by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/768
- Introduce a New Parquet Write Strategy & Support read configuration to enable parquet by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/769
- FIX [exchange_client] Clean up hashmap when ringbuffer is full by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/770
- Add parquet find related function. by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/771
- Finish aio in synchronous to free smsg when flushing is done by @wanghaEMQ in https://github.com/nanomq/NanoNNG/pull/772
- Parquet support aes encryption." by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/773
- new conf option to disable default tcp by @OdyWayne in https://github.com/nanomq/NanoNNG/pull/775
- MQ: Exchange: Key value of msg is stored in stack space by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/774
- MDF [exchange_client] modify threading model of exchange by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/778
- NEW [conf_ver2] disable tcp by not config it by @OdyWayne in https://github.com/nanomq/NanoNNG/pull/777
- MQ: exchange: Introduce sock_recv for get_msgs from exchange by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/776
- Fixed parquet find before init error. by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/780
- FIX [conf] Support multiple exchange_client by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/781
- MDF [conf_ver2] add enable option to parquet by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/782
- FIX [exchange] fix wrong parser of conf ver by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/784
- Fix the heap over flow on parquet queue and update some logs by @wanghaEMQ in https://github.com/nanomq/NanoNNG/pull/785
- Support conf encryption option. by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/786
- Sync upstream by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/779
- Use parquet static lib. by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/787
- Use uint64_t as key of MQ by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/789
- Change key from uint32 to uint64 (parquet). by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/788
- MDF [parquet] fix uint64_t key of parquet file name by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/790
- FIX [exchange] Remove some outdate logs. by @wanghaEMQ in https://github.com/nanomq/NanoNNG/pull/791
- MDF [parquet] More stable get_file_name by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/793
- enable override retain flag by @OdyWayne in https://github.com/nanomq/NanoNNG/pull/794
- MDF [conf] enable bridge retain flag in old conf by @OdyWayne in https://github.com/nanomq/NanoNNG/pull/795
- Add listener+cb as first version of consumer SDK by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/797
- New fuzz search api for exchange_server and ringbuffer by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/798
- Refactor configuration of exchange by @wanghaEMQ in https://github.com/nanomq/NanoNNG/pull/796
- Sync with upstream & fix confilcts by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/800
- Add variety safety checker for logging by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/802
- Exchange server support fuzz search request from consumer(ringbuffer and parquet) by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/803
- Exchange consumer demo by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/804
- Support fuzz search request in sock_recv by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/805
- Speeding up message composition by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/809
- Fix #806 by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/807
- old conf alloc by cvector by @OdyWayne in https://github.com/nanomq/NanoNNG/pull/810
- Sync changes in tcp transport layer (broker and sdk) to tls. by @wanghaEMQ in https://github.com/nanomq/NanoNNG/pull/808
- Support read plugin from config file by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/811
- Fixed type error. by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/814
- plugin need SUPP_PLUGIN instead of SUPP_RULE_ENGINEE by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/815
- FIX [exchange_server] Initial alloc memory as 0, otherwise ASAN will coredump when strlen by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/813
- Rewrite parquet_find_span function with start key, end key as paramters && polish queue FOR_EACH macro. by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/816
- NEW [conf] add new field total_ctx in conf by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/819
- User-defined operation types are supported when rb is full by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/818
- more tcp prarms for bridge by @OdyWayne in https://github.com/nanomq/NanoNNG/pull/817
- fix https://github.com/nanomq/nanomq/issues/1619 by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/820
- Clean up some code warning by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/821
- New parquet write function. by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/822
- remove log & add disconnect decoding by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/823
- fix the bug https://github.com/nanomq/nanomq/issues/1619 for tls/ws by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/825
- FIX [util] fix warnings by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/827
- Add parquet_file_range && replace cb with aio. by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/824
- Update parquet_object. by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/828
- Fix #812 by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/826
- Fix some errors in parsing auth.http config. by @wanghaEMQ in https://github.com/nanomq/NanoNNG/pull/833
- Minor fixes of warnings & modify get_Var_integar by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/832
- Support parquet find packets function. by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/831
- [ringbuffer] Write msgs to file with parquet and support dumping all messages in file by @RanMaoyi in https://github.com/nanomq/NanoNNG/pull/830
- Reduce ineffective file_ranges passing and improve the performance of ringbus lookups from files by @lee-emqx in https://github.com/nanomq/NanoNNG/pull/834
- Bump NNG version to 1.7.1 & add new hash64 API by @JaylinYu in https://github.com/nanomq/NanoNNG/pull/835 ## New Contributors
- @lkyjason made their first contribution in https://github.com/nanomq/NanoNNG/pull/755 Full Changelog: https://github.com/nanomq/NanoNNG/compare/0.20.8...0.21.0