You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
The broker quit operation - "Keep alive time overdue". #35
In an attempt to isolate the problem I set up a separate system for one device measuring, a broker and a client only listening. After some time I intentionally reboot the device. After that it does not seem to (re)connect, I do not know the cause for that. But I do not expect the broker to quit because of that.
Ohh, I see. Your error message is also The remote computer refused the network connection, that makes me think it's an OS/device error instead of a nmqtt error which just is Connection refused. Could it be some kind of user-restriction, requirement of manual activation, etc. on the device? I have no experience with Shelly devices, so I can't guide you on that one. Also you might want to give the keep alive time a look.
Pulling the whole device from the power socket so it stops sending data has the same result. After some time the broker quits.
Another situation works fine. I start the broker and sub_client. After that I put the device in the socket. It connects fine and sends the data as to be expected.
Without going through the whole process and would probably understand half of it I noticed: In keepAliveMonitor state is set to error. Then close() is called. close() does not check for error state.
Tested it today with Mosquitto, nmqtt_sub and the Shelly. No problem. I can reset the device, it goes off-line, comes back on-line and everything continues as it should. Same when I just pull the plug and put it back.
Also tested it with HBMQTT broker (python). Same result, it works flawless.
Can you replicate it, if you only subscribe to a single topic per client?
Subscribing to: shellies/shellyplug-s-BCFF4D5A2607/relay/0
Subscribing to: shellies/shellyplug-s-BCFF4D5A2607/relay/0/energy
Subscribing to: shellies/shellyplug-s-BCFF4D5A2607/relay/0/power
Subscribing to: shellies/shellyplug-s-BCFF4D5A2607/relay/0/overpower_value
Subscribing to: shellies/shellyplug-s-BCFF4D5A2607/temperature
Subscribing to: shellies/shellyplug-s-BCFF4D5A2607/overtemperature
Could you try to clone the repo and change the output message? Then we can inspect the client data.
I cannot recreate the scenario, so I can't force the error. But it could sound like, that the broker does not recognize that the client has disconnected, and then errors out on a dead client.
A way to recreate the problem I tested today: run a listener client on a laptop connected with wifi, run the broker on an other machine. Then turn off the wifi connection on the laptop. After ~ 60 sec the broker will quit.
in nmqtt.nim, inside processClient I wondered how sendWill if diconnected? So I changed it to removeSubscriber as it has disconnected from the broker:
if ctx.state == Error and ctx.beenConnected:
# This happens on a ungraceful disconnects from the client.
#asyncCheck sendWill(ctx)
asyncCheck removeSubscriber(ctx)
I'm still testing things. It is a strange problem and I have trouble following the flow. In what I did no Will will be send at all, that's not a proper solution. For some reason with the will part in it the server blips out of existance at the end of processClient.
A little step further, sendWill proc starts of with a test against ctx.willTopic. Should it not test against ctx.willFlag as the topic can have content but one does not want to send it. This is the case with the Shellies:
when defined(broker):
proc sendWill(ctx: MqttCtx) {.async.} =
## Send the will
if ctx.willTopic != "": ### should be if ctx.willFlag:
for c in mqttbroker.subscribers[ctx.willTopic]:
let msgId = c.nextMsgId()
let qos = qosAlign(ctx.willQos, c.subscribed[ctx.willTopic])
c.workQueue[msgId] = Work(wk: PubWork, msgId: msgId, topic: ctx.willTopic, qos: qos, message: ctx.willMsg, typ: Publish)
await c.work()
Now, after this change unplugging the device seems to works flawless. More testing needed.
But now the question is why does the server crash without the change?? There is a topic and there is a message that can be send, regardless of the flag. Is the something wrong in the sending of the LWT?
I wrote two clients. On standard one testclient1 where I set the will. I notice that in the server stdoutput that there is a will topic and message, but that the willFlag == false. After 20 s the client is made to crash and the server follows soon and crashes too.
In the second client I added a modified 'set_will' that sets a will topic and message but with a willFlag == false. I notice that in the server stdoutput that there is no will topic or message and that the willFlag == false. After 20 s the client is made to crash, the server continues as it should.
As I do not understand the flow and big chunks of (n)mqtt that's about what I can figure out. The problem seems to be connected with the willFlag somehow.
testclient1.nim :
import std/[times, strformat]
from os import getCurrentProcessId
include nmqtt
let pid = getCurrentProcessId()
let ctx = newMqttCtx(fmt"something_sub_{pid}")
ctx.set_host("192.168.1.11", 1883)
ctx.set_will("something/lwt","offline") #defaults to willFlag = true
ctx.set_verbosity(3)
proc mqttSub() {.async.} =
await ctx.start()
proc on_data(topic: string, message: string) =
echo $getTime().utc.format("yyyy-MM-dd'T'HH:mm:ss'.'fff"),";", topic, ";", message
await ctx.subscribe("something/from/somewhere", 2, on_data)
var a:ptr[int]
waitFor sleepAsync(20000) # wait 20 seconds before destruction
echo a[]
asyncCheck mqttSub()
runForever()
testclient2 :
import std/[times, strformat]
from os import getCurrentProcessId
include nmqtt
proc set_falsewill*(ctx: MqttCtx, topic, msg: string, flag=false, qos=0, retain=false) =
## Set the clients will.
ctx.willFlag = flag
ctx.willTopic = topic
ctx.willMsg = msg
ctx.willQoS = qos.uint8
ctx.willRetain = retain
let pid = getCurrentProcessId()
let ctx = newMqttCtx(fmt"something2_sub_{pid}")
ctx.set_host("192.168.1.11", 1883)
ctx.set_falsewill("something/lwt","offline") #defaults to willFlag = false
ctx.set_verbosity(3)
proc mqttSub() {.async.} =
await ctx.start()
proc on_data(topic: string, message: string) =
echo $getTime().utc.format("yyyy-MM-dd'T'HH:mm:ss'.'fff"),";", topic, ";", message
await ctx.subscribe("something/from/somewhere", 2, on_data)
var a:ptr[int]
waitFor sleepAsync(20000) # wait 20 seconds before destruction
echo a[]
asyncCheck mqttSub()
runForever()
The following changes seem to fix the problem as far as I can test. Please test.
First the willFlag had not been set in the context, adding the ctx.willFlag = true in proc onConnect(ctx: MqttCtx, pkt: Pkt) fixes that. Line 687, 688.
fixed version:
proc onConnect(ctx: MqttCtx, pkt: Pkt) {.async.} =
when not defined(broker):
ctx.wrn "Packet type only supported for broker: " & $pkt.typ
else:
var
offset: int
nextLen: uint16
# Main data
(ctx.proto, offset) = pkt.getstring(0, true)
(ctx.version, offset) = pkt.getu8(offset)
(ctx.connFlags, offset) = getbin(pkt, offset)
(ctx.keepAlive, offset) = pkt.getu16(offset)
(nextLen, offset) = pkt.getu16(offset)
(ctx.clientId, offset) = pkt.getstring(offset, parseInt($nextLen))
if not mqttbroker.spacesInClientId:
ctx.clientid = ctx.clientid.replace(" ", "")
# Will Topic
if ctx.connFlags[5] == '1':
ctx.willFlag = true <--------------------- this line added
(nextLen, offset) = pkt.getu16(offset)
(ctx.willTopic, offset) = pkt.getstring(offset, parseInt($nextLen))
(nextLen, offset) = pkt.getu16(offset)
(ctx.willMsg, offset) = pkt.getstring(offset, parseInt($nextLen))
etc.
Then sending a will fails and crashes the server if there are no subscribers to it:
current version:
when defined(broker):
proc sendWill(ctx: MqttCtx) {.async.} =
## Send the will
if ctx.willTopic != "":
for c in mqttbroker.subscribers[ctx.willTopic]:
let msgId = c.nextMsgId()
let qos = qosAlign(ctx.willQos, c.subscribed[ctx.willTopic])
c.workQueue[msgId] = Work(wk: PubWork, msgId: msgId, topic: ctx.willTopic, qos: qos, message: ctx.willMsg, typ: Publish)
await c.work()
fixed version:
when defined(broker):
proc sendWill(ctx: MqttCtx) {.async.} =
## Send the will
if ctx.willFlag:
if mqttbroker.subscribers.hasKey(ctx.willTopic):
for c in mqttbroker.subscribers[ctx.willTopic]:
let msgId = c.nextMsgId()
let qos = qosAlign(ctx.willQos, c.subscribed[ctx.willTopic])
c.workQueue[msgId] = Work(wk: PubWork, msgId: msgId, topic: ctx.willTopic, qos: qos, message: ctx.willMsg, typ: Publish)
await c.work()
Asked Claude about both proposed fixes above fwiw:
Both fixes look correct and well-reasoned. Here's a breakdown:
Fix 1 — Setting ctx.willFlag = true in onConnect
This is a real bug. The broker parses the will topic/message from the connect packet when connFlags[5] == '1', but never sets ctx.willFlag. That field was only ever set via the client-side set_will() proc. So the broker's own ctx objects for remote clients never had willFlag set, making any check on it useless. This fix closes that gap.
Fix 2 — Guarding sendWill with hasKey
This is the crash fix. The original:
if ctx.willTopic !="":
for c in mqttbroker.subscribers[ctx.willTopic]: # KeyError if no subscribers!
directly indexes into the table without checking existence. If a client registered a will but nobody subscribed to that topic, the broker crashes with a KeyError on disconnect. The proposed fix correctly guards with both ctx.willFlag (now reliable thanks to Fix 1) and hasKey.
One minor note: the original sendWill also doesn't check hasKey for c.subscribed[ctx.willTopic] inside the loop — if a subscriber somehow lost that topic entry, it would also crash. Worth adding a guard there too:
procsendWill(ctx: MqttCtx) {.async.} =if ctx.willFlag and mqttbroker.subscribers.hasKey(ctx.willTopic):
for c in mqttbroker.subscribers[ctx.willTopic]:
ifnot c.subscribed.hasKey(ctx.willTopic): continue# extra safetylet msgId = c.nextMsgId()
let qos =qosAlign(ctx.willQos, c.subscribed[ctx.willTopic])
c.workQueue[msgId] =Work(wk: PubWork, msgId: msgId, topic: ctx.willTopic, qos: qos, message: ctx.willMsg, typ: Publish)
await c.work()
Also worth double-checking that sendWill is actually being called in onDisconnect — looking at the code it's commented out there (#await sendWill(ctx)), so you'll want to uncomment that for the will to fire at all.
Activity
ingoogni commentedon Sep 27, 2022
In an attempt to isolate the problem I set up a separate system for one device measuring, a broker and a client only listening. After some time I intentionally reboot the device. After that it does not seem to (re)connect, I do not know the cause for that. But I do not expect the broker to quit because of that.
The broker:
A client just listening to the broker fwiw:
ThomasTJdev commentedon Sep 28, 2022
Hi @ingoogni
Thanks for reporting it - it sounds strange. I have tested the setup without any problems:
When the broker disconnects, the clients will try to auto-reconnect like this:
ingoogni commentedon Sep 28, 2022
Thanks for looking in to it.
When I simulate it like you do, I get the same results as you do.
The problem that occurs with the rebooting of the Shelly device is reproducible.
The Shelly device is a commercial one, build on an ESP8266 with Mongoose OS. That's all the info I have on that.
It will take some time, I can try the device with an other broker if that helps.
ThomasTJdev commentedon Sep 28, 2022
Ohh, I see. Your error message is also
The remote computer refused the network connection, that makes me think it's an OS/device error instead of anmqtterror which just isConnection refused. Could it be some kind of user-restriction, requirement of manual activation, etc. on the device? I have no experience with Shelly devices, so I can't guide you on that one. Also you might want to give thekeep alive timea look.ingoogni commentedon Sep 28, 2022
That is the nmqtt subscriber client. It does that when the broker is not online. Once the broker is on line again it connects fine.
There is something I have not tried. It is a plug so I can just pull it from its socket, wait and see what happens.
ingoogni commentedon Sep 28, 2022
Pulling the whole device from the power socket so it stops sending data has the same result. After some time the broker quits.
Another situation works fine. I start the broker and sub_client. After that I put the device in the socket. It connects fine and sends the data as to be expected.
Broker log of pulling the plug:
nmmqt_sub client for plotting the values:
ingoogni commentedon Sep 28, 2022
Without going through the whole process and would probably understand half of it I noticed: In
keepAliveMonitorstate is set toerror. Thenclose()is called.close()does not check forerrorstate.ingoogni commentedon Sep 29, 2022
Tested it today with Mosquitto, nmqtt_sub and the Shelly. No problem. I can reset the device, it goes off-line, comes back on-line and everything continues as it should. Same when I just pull the plug and put it back.
Also tested it with HBMQTT broker (python). Same result, it works flawless.
Hope it helps.
ingoogni commentedon Oct 8, 2022
When I stop (ctrl-C or taskmanager) a nmqtt client, the broker prints
The after ~60 seconds the broker prints:
Connections >> nmqtt_client_name was disconnected. Keep alive time overdue.and continues as usual.
When I pull a shelly plug, or reboot it the
message from the broker is not there. The power to the client was cut so no "goobye message" could have been sent.
Could this make the difference?
ThomasTJdev commentedon Oct 9, 2022
Can you replicate it, if you only subscribe to a single topic per client?
Could you try to clone the repo and change the output message? Then we can inspect the client data.
I cannot recreate the scenario, so I can't force the error. But it could sound like, that the broker does not recognize that the client has disconnected, and then errors out on a dead client.
ingoogni commentedon Oct 9, 2022
A way to recreate the problem I tested today: run a listener client on a laptop connected with wifi, run the broker on an other machine. Then turn off the wifi connection on the laptop. After ~ 60 sec the broker will quit.
Below the output of the patched broker:
ingoogni commentedon Feb 12, 2024
in
nmqtt.nim, insideprocessClientI wondered howsendWillif diconnected? So I changed it toremoveSubscriberas it has disconnected from the broker:and all seems to work fine now...?
ThomasTJdev commentedon Feb 15, 2024
Hi @ingoogni
Looks good. I'll schedule some tests before approving the PR. Maybe @srd424 will pitch in on the testing?
ingoogni commentedon Feb 15, 2024
I'm still testing things. It is a strange problem and I have trouble following the flow. In what I did no
Willwill be send at all, that's not a proper solution. For some reason with the will part in it the server blips out of existance at the end ofprocessClient.ingoogni commentedon Feb 15, 2024
A little step further,
sendWillproc starts of with a test againstctx.willTopic. Should it not test againstctx.willFlagas the topic can have content but one does not want to send it. This is the case with the Shellies:Now, after this change unplugging the device seems to works flawless. More testing needed.
But now the question is why does the server crash without the change?? There is a topic and there is a message that can be send, regardless of the flag. Is the something wrong in the sending of the LWT?
ingoogni commentedon Feb 16, 2024
Tested with a pristine install of nmmqt on win11.
I wrote two clients. On standard one
testclient1where I set the will. I notice that in the server stdoutput that there is a will topic and message, but that thewillFlag == false. After 20 s the client is made to crash and the server follows soon and crashes too.In the second client I added a modified 'set_will' that sets a will topic and message but with a
willFlag == false. I notice that in the server stdoutput that there is no will topic or message and that thewillFlag == false. After 20 s the client is made to crash, the server continues as it should.As I do not understand the flow and big chunks of (n)mqtt that's about what I can figure out. The problem seems to be connected with the willFlag somehow.
testclient1.nim:testclient2:testclient1.exeoutputtestclient2is simmilar:server output serving
testclient1.exe:server output serving
testclient2.exe:ingoogni commentedon Feb 16, 2024
it seems to met that the
willFlagis not set inproc onConnect(ctx: MqttCtx, pkt: Pkt). Is that proper?ingoogni commentedon Feb 20, 2024
The following changes seem to fix the problem as far as I can test. Please test.
First the
willFlaghad not been set in the context, adding thectx.willFlag = trueinproc onConnect(ctx: MqttCtx, pkt: Pkt)fixes that. Line 687, 688.fixed version:
Then sending a will fails and crashes the server if there are no subscribers to it:
current version:
fixed version:
ingoogni commentedon Apr 27, 2026
Asked Claude about both proposed fixes above fwiw:
Both fixes look correct and well-reasoned. Here's a breakdown:
Fix 1 — Setting ctx.willFlag = true in onConnect
This is a real bug. The broker parses the will topic/message from the connect packet when connFlags[5] == '1', but never sets ctx.willFlag. That field was only ever set via the client-side set_will() proc. So the broker's own ctx objects for remote clients never had willFlag set, making any check on it useless. This fix closes that gap.
Fix 2 — Guarding sendWill with hasKey
This is the crash fix. The original:
directly indexes into the table without checking existence. If a client registered a will but nobody subscribed to that topic, the broker crashes with a KeyError on disconnect. The proposed fix correctly guards with both ctx.willFlag (now reliable thanks to Fix 1) and hasKey.
One minor note: the original sendWill also doesn't check hasKey for c.subscribed[ctx.willTopic] inside the loop — if a subscriber somehow lost that topic entry, it would also crash. Worth adding a guard there too:
Also worth double-checking that sendWill is actually being called in onDisconnect — looking at the code it's commented out there (#await sendWill(ctx)), so you'll want to uncomment that for the will to fire at all.