Mikrotik LHG(G) - secondary CA B3 band auto reconnect

[LeaUK]

Hi all

Mikrotik LHG(G)
Router OS v7.12

I note one of my CA B3 drops occasionally and the LTE modem doesn't auto re-negotiate to bring this secondary channel back on-line. I also note regular (every 5 seconds or so) AT command chatter to and from the mast, but last night this stopped (maybe coincidence?).

Anyone know how to (script?) auto detect a dropped secondary CA band and auto re-negotiate with the cell, preferably without disable/enable the interface as this will interrupt any connections?

Cheers
Lea

 

[green]

Additional Component Carriers are added into aggregation on demand, according to the link utilization. Run a speed test and see if there is a change.

 

[Norfolk&Signal]

^--- What 'green' said

 

[LeaUK]

Thanks, I have, multiple times and also with large file downloads, no change, hence reaching out.

Upon initial negotiation the secondary carrier is always added, however it seems once it drops (for unknown reasons), it's never added back - 1.5 days and counting.

I'm wondering why the constant AT chatter has disappeared too?

Does anyone else see this?

 

[green]

AT command chatter to and from the mast

Your router can talk AT to your modem only. My understanding is that you can see AT messages in the router log, perhaps this will help understand the nature of such activity.

 

[LeaUK]

Cheers, I'm logging AT messages and can see rx and tx, and from 1am a few days ago they stopped, which is odd, and B3 secondary dropped. I'm wondering if they're linked.

I'm confident if I disable and enable the LTE (modem) primary and secondary will be negotiated, but I want to understand why this has occurred and how to reestablish the CA automatically, maybe by script, however preferably without reinitialising the modem as this will drop all connections.

I need to go read up on some CLI - https://wiki.mikrotik.com/wiki/Manual:Interface/LTE

 

[LeaUK]

Uhmmmm, new to Mikrotik, school boy error (FTFM)? I didn't update the RouterBOARD/BOOT firmware to match RouterOS.

When using a RouterBOARD device, it is always suggested to upgrade its RouterBOOT bootloader after RouterOS is upgraded. To do this, issue the command "/system routerboard upgrade"

and

After updating the MikroTik RouterOS, it is always recommended to update the firmware of the MikroTik RouterBOARD.

Then restart.

Can also be completed via WinBox > System > RouterBOARD

So now the board firmware and RouterOS are at v7.12.

Of course both B3 primary and CA secondary B3 established as expected on a restart, and LTE AT commands (chatter every 5 seconds) are being logged once again. Now to see if (when) the secondary B3 drops out.

Couldn't see much in the docs about renegotiating the connection without dropping, booo! MIght have to script a reconnect on an overnight scheduler alongside a regular ping check for internet as I've noticed on occasion the operator drops, and the LTE fails to reconnect (that's another story).

 

[AnotherGig]

Don't want to blow your mind, but assuming that your LHGG still contains the original R11e-LTE6 modem, you can also check for / initiate a firmware update for the modem as well:

LTE - RouterOS - MikroTik Documentation

help.mikrotik.com

Have fun!

 

[LeaUK]

Cheers, good tip, but yep already checked (thankfully Winbox informed me) and updated from R11e-LTE6_V025 to R11e-LTE6_V036

Although the CLI fails with error on command: /interface lte info lte1 once or /interface lte1 info lte1 once

Not sure why yet.

 

[LeaUK]

So again at around 12:01am the AT chatter stops and I find my B3 secondary missing and it wont renegotiate without a LTE reset - annoying!

 

[LeaUK]

Additional Component Carriers are added into aggregation on demand, according to the link utilization. Run a speed test and see if there is a change.

Sadly I can't reproduce this, any further tips appreciated.

Cheers

 

[LeaUK]

Update:

After an evening / morning of investigation, it appears EE disable the secondary B3 cell overnight (power saving?) and sadly the modem doesn't renegotiate the link (CA) at any point after.

Here's the sequence of events:

1. On initial connection, LTE6 (CA modem) negotiates primary B3 & secondary B3 CA
2, At 12am, EE network carrier disables secondary B3 (perhaps power saving over night)
3, LTE6 AT chatter stops as shown in lack of logging information
4. LTE6 displays no CA and remains connected to B3 primary only, significantly reducing bandwidth
5. At 7am, EE enables secondary B3 cell.
6. LTE6 modem does not renegotiate connection and thus remains attached to primary B3 ONLY. Only half the bandwidth is available throughout the working day. LTE6 never renegotiates leaving the modem without CA.
L7. LTE6 will renegotiate both primary B3 and secondary B3 (CA) if LTE6 interface is disabled/enabled after 7am (and before 12am) or device rebooted, but this is not automated by modem or RouterOS.

 

[SA20]

Is this any help…

How to Automatically Reboot Your MikroTik Router

 

[GavinAshford]

I did suspect it could be the B3 second carrier overnight power saving, but not having any experience with mikrotik/AT I didn't couldn't explain if the lack of AT chatter was a cause or a consequence.

 

[LeaUK]

You would be correct. The AT chatter is directly correlated to the secondary CA, hence it stops after the cell goes off-line.

A simple scheduled script to disable / re-enable the LTE should do it, and quickly in comparison to full reboot (that takes longer than my YouTube buffer.

Create a script named LTE:

/interface lte disable lte1
:delay 3s;
/interface lte enable lte1

Create a scheduler item:
/system scheduler add name="LTE" on-event=LTE interval=1d start-time=07:15:00

Not the most elegant, but still, skinning cats and all that!

Cheers all

 

[Norfolk&Signal]

If that doesn't work you could band switch in a scheduled script. Set it to B20 only and then switch it back to B3.

I do a similar 'perk up' script on my LHG on a heavily congested Three mast. I have to switch it to B20 only, pause, add B3 (B20+B3 CA) and then I set it to B3 only.

I also get the script to write entries into the logs for reference

:log info "LTE-BAND-TURBO Script Started" :delay 5 :log info "NAS-TOGGLE-LTE drop" [/interface disable lte1] :delay 5 [/interface enable lte1] :log info "NAS-TOGGLE-LTE up" [/interface lte set lte1 band=20] :log info "LTE-BAND-TURBO set to B20" :delay 60 [/interface lte set lte1 band=20,3] :log info "LTE-BAND-TURBO set to B20,3" :delay 30 :log info "LTE-BAND-TURBO setting to B3" [/interface lte set lte1 band=3] :log info "LTE-BAND-TURBO Script Completed"

 

[LeaUK]

Cheers, useful to have some script examples as I'm writing similar although testing shows on my local mast no need to band switch, CA is restored. However, (and worryingly), on one occasion an unexpected issue occurred on enable, new IP OK, link up, both B3s (CA) BUT no internet, Internet Detect shows orange (partial) connection and no internet routing available to client. Odd, never seen this before, I've increased the delay between disable and enable from 3 to 5 seconds as a precautionary.

Also (with kind assistance) wrote a check for IP script which I may run every 5 mins, if ping fails it will auto disable/enable LTE interface. Haven't decided to run this continuously until testing concludes (a little concerned around race conditions) or I may simply run post am mast reconnect script, as I've never seen internet actually drop. Running every 5 mins may actually introduce issues...?

Running it every 5 mins today to watch for side effects.

One last thought, another kind contact pointed me to the command: /interface lte at-chat [find] input="AT+RESET" rather than disable/enable - this forces a full LTE and network channel restart and I've seen this is required on occasion over disable/enable - such as a change of APN type/creds, BUT it takes far longer to affect (longer than a typical YouTube buffer) which may become intrusive of internet use.

For those creating MikroTik scripts, a handy syntax checking tool is to create a script and point to it in CLI to check for error, look for red cursor:

/system script print without-paging where name="<scriptname>"

Anyone have experience of /interface lte at-chat [find] input="AT+RESET" ?

Cheers
Lea

 

[Norfolk&Signal]

Top tip with the IP check script - use hostname not IP. I have a similar 'check google' script that I have installed on my LHG plus all my customer installs - and that checks for 'google.com' rather than, say, 8.8.8.8.

Pinging 8.8.8.8 won't flag up an issue with DNS, but pinging 'google.com' will. This is the voice of experience as my first iteration of my 'ping test' script checked IP address 8.8.8.8 and it didn't pick up on a DNS issue on a customer install - they had internet but no DNS - lesson learned, script upgraded, no problems since. I schedule it to run it every 15 minutes - I see little point running it more frequently than that. This also allows me a 15 minute window for messing about with stuff without triggering the disconnect/reconnect when I don't want it to.

 

[green]

You can try to toggle the radio only, using standard commands
at+cfun=4
wait a moment
at+cfun=1

 

[LeaUK]

Top tip with the IP check script - use hostname not IP. I have a similar 'check google' script that I have installed on my LHG plus all my customer installs - and that checks for 'google.com' rather than, say, 8.8.8.8.

Pinging 8.8.8.8 won't flag up an issue with DNS, but pinging 'google.com' will. This is the voice of experience as my first iteration of my 'ping test' script checked IP address 8.8.8.8 and it didn't pick up on a DNS issue on a customer install - they had internet but no DNS - lesson learned, script upgraded, no problems since. I schedule it to run it every 15 minutes - I see little point running it more frequently than that. This also allows me a 15 minute window for messing about with stuff without triggering the disconnect/reconnect when I don't want it to.

Indeed, I'm using google.co.uk based on it's going to be more resilient than even Mikrotik's inbuilt 'Internet Detect' mechanism of UDP to cloud.mikrotik, wouldn't want my internet bouncing based on Mikrotik hosting issues.