esp32

We will focus on Home Assistant with an install base of over 100,000 user

An open source home automation that puts local control and privacy first. Powered by a worldwide community of tinkerers and DIY enthusiasts. Perfect to run on a Raspberry Pi or a local server.

Home Assistant

We auto populate the Home Assistant entities using MQTT Discovery. This enables us to use MQTT devices with only minimal configuration effort on the side of Home Assistant. A few of these messages are shown in the section MQTT device discovery of chapter Interface.

Home Assistant supports will use the messages and automatically populate the entities shown below.

Dashboard

In the Home Assistant configuration.yaml we add some rules and template sensors as shown in hassio/packages. This also includes some rules that we use for the PoolMath integration.

The Lovelace dashboard tab includes several cards as shown below. The YAML code can be found in hassio/lovelace.

Idle

When the pump is not running, the cards shown below wil be visible.

When for instance the pool circuit is activated, additional cards show up.

Other home automation systems

The OPNpool device publishes state transitions using MQTT. You find an overview of the topics in the table below.

{
    "mode": "heat",
    "heatsrc": "None", // None/Heater/SolarPref/Solar
    "target_temp": 70,
    "current_temp": 67,
    "action": "off"  // off/heating/idle
}

{
    "mode": "heat",
    "heatsrc": "None", // None/Heater/SolarPref/Solar
    "target_temp": 0,
    "current_temp": 67,
    "action": "off"  // off/heating/idle
}

When the OPNpool device connects to the WiFi access point, it will be assigned an IP address. Only the access point and the device know this address. Other hosts can access the device by its mDNS name opnpool.local.

This relies on service auto-discovery using the Multicast DNS (mDNS) protocol. Note that mDNS’ use of multicast packets is designed to work within a single IP subnet. That implies that in general your computer/phone need to be on the same subnet as the OPNpool device for the mDNS protocol to work.

HTML

As you may have seem in the video in the Deploying chapter, the OPNpool device can be controlled through a simple HTML UI. This allows the user to monitor the state of the pool controller, pump and chlorinator.

This UI is accessed through the URL http://opnpool.local/. The unsecure connection implies that security solitary relies on your WiFi credentials. If you want to access the device from the public internet, I urge you by tunneling into your LAN instead of simply opening a port on your router.

To conserve memory, the OPNpool device only contains the minimum HTML code that refers to an external site for the Stylesheet and the replacement element that refers to the JavaScript.

Some examples of the UI are shown below

JSON

The raw state of the pool controller is accessible through http://opnpool.local/json. The HTML page uses this internally, but you can also poll this state from home automation systems. However, if your home automation supports MQTT, I suggest using MQTT as described in the Home Assistant chapter.

Who

The IP address of the OPNpool device can be found using the http://opnpool.local/who page. An example is shown in illustration below.

The interface image makes the device an “OPNpool Interface”. Besides monitoring the pool controller and providing access to a web interface, it provides automatic updates, network discovery and supports post mortem debugging.

The entry point of the code is app_main() in the file main/main.c.

app_main() starts by initializing devices and connecting to the WiFi network. The interface appplication is split over different tasks that communicate over message queues offered by the Free Real Time Operating System (Freitas).

• factory_reset_task
• pool_task
• ota_update_task
• http server
• mqtt_task
• hass_task

factory_reset_task

A factory reset erases the Wifi credentials and returns to the factory app. This allows the device to be provisioned from scratch. To perform a Factory Reset, pull GPIO#0 low for five seconds, while the interface app is running (not during boot up). The development board, has a boot button) that services the same function. I

                I (753) factory_reset_task: Waiting for 3 sec BOOT/RESET press ..
                W (26408) factory_reset_task: Factory reset ..
            

The code base uses the git submodule ESP32_factory-reset-task.

pool_task

The pool_task forms the heart of OPNpool. It handles data received from the pool controller, and can send requests to the pool controller.

Its responsibilities are:

• Initially, it queues a few messages that request thermostat and scheduling information from the pool controller.
• If it receives a message in the to_pool_q, it creates a corresponding network message. It then calls datalink_tx_pkt_queue to add the preamble and tail, and queues it for transmission through the rs485 driver.
• Reads bytes from the rs485 driver to create a data packet. It passes this up the stack create a network message. Based on that it updates the pool state information and publishes it using the MQTT task.

The protocol stack is shown in the illustration below.

The following sections provide more information about the protocol stack.

RS-485 driver

Starting from the physical RS-485 bus, the signal passes through the MAX3485 transceiver that connects to the RX, TX and RTS pins of the ESP32 UART. This UART presents the signal as bytes to the driver.

The driver:

• Initializes the UART.
• Provides functions to read and write bytes.
• Bytes to be transmitted are queued as as rs485_q_msg in rs485->tx_q. When a transmit opportunity arrises, the pool_task dequeues from rs485->tx_q, relies on the driver to transmits the bytes.

Transmitting on the RS-485 is a bit tricky because it operates in half-duplex and has multiple nodes on the same single twisted pair. In practice, the pool controller appears to only accept messages after it broadcasts its periodic status message, and before it communicates with the chlorinator. Therefore, outgoing bytes are queued, until the moment arrives where they can be transmitted

It communicates with the datalink layer using the rs485_handle_t device handle.

        typedef struct rs485_instance_t {
            rs485_available_fnc_t available;      // bytes available in rx buffer
            rs485_read_bytes_fnc_t read_bytes;    // read bytes from rx buffer
            rs485_write_bytes_fnc_t write_bytes;  // write bytes to tx buffer 
            rs485_flush_fnc_t flush;              // wait until all bytes are transmitted
            rx485_tx_mode_fnc_t tx_mode;          // controls RTS pin (for half-duplex)
            rs485_queue_fnc_t queue;              // queue to handle->tx_q
            rs485_dequeue_fnc_t dequeue;          // dequeue from handle->tx_q
            QueueHandle_t tx_q;                   // transmit queue
        } rs485_instance_t;
        typedef struct rs485_instance_t * rs485_handle_t;
    

Datalink layer

Provides framing and byte error detection. The interface with the network layers is through datalink_pkt_t.

It supports both the A5 and IC protocol.

It communicates with the network layer using the datalink_pkt_t structure.

        typedef struct datalink_pkt_t {
            datalink_prot_t    prot;      // datalink_prot as detected by `_read_head`
            uint8_t            prot_typ;  // from datalink_hdr_a5->typ
            uint8_t            src;       // from datalink_hdr_a5->src
            uint8_t            dst;       // from datalink_hdr_a5->dst
            datalink_data_t *  data;
            size_t             data_len;
            skb_handle_t       skb;
        } datalink_pkt_t;
    

Its functions are:

• datalink_rx_pkt(), receives bytes from the RS-485 bus and makes a datalink_pkt from it. It is implemented as a state machine, using socket buffers to store the bytes received.
• datalink_tx_pkt(), receives a datalink_pkt and wraps it with a header and tailer. It then queues it for transmission on the RS-485 bus. Once there is a transmit opportunity, the pool_task send it using the rs485 driver.

Network layer

This layer translates between datalink_pkt and the protocol-agnostic network_msg.

It communicates with the `poolstate` layer using the network_msg_t structure.

        typedef struct network_msg_t {
            network_msg_typ_t typ;
            union {
                network_msg_pump_reg_set_t * pump_reg_set;
                network_msg_pump_reg_resp_t * pump_reg_set_resp;
                network_msg_pump_ctrl_t * pump_ctrl;
                network_msg_pump_mode_t * pump_mode;
                network_msg_pump_run_t * pump_run;
                network_msg_pump_status_resp_t * pump_status_resp;
                network_msg_ctrl_set_ack_t * ctrl_set_ack;
                network_msg_ctrl_circuit_set_t * ctrl_circuit_set;
                network_msg_ctrl_sched_resp_t * ctrl_sched_resp;
                network_msg_ctrl_state_t * ctrl_state;
                network_msg_ctrl_time_t * ctrl_time;
                network_msg_ctrl_heat_t * ctrl_heat;
                network_msg_ctrl_heat_set_t * ctrl_heat_set;
                network_msg_ctrl_layout_t * ctrl_layout;
                network_msg_ctrl_layout_set_t * ctrl_layout_set;
                network_msg_chlor_ping_req_t * chlor_ping_req;
                network_msg_chlor_ping_resp_t * chlor_ping;
                network_msg_chlor_name_t * chlor_name;
                network_msg_chlor_level_set_t * chlor_level_set;
                network_msg_chlor_level_resp_t * chlor_level_resp;
                uint8_t * bytes;
            } u;
        } network_msg_t;
    

Its public functions are:

• network_rx_msg(), translates a datalink_pkt writes it as network_msg.
• network_create_msg(), creates a datalink_pkt from a network_msg.

Poolstate

The sysState_t structure stores the state of the pool system. It is updated based on status messages from the controller, and by querying the controller. Once the controller replied to these queries, pool() starts honoring commands from the Serial port connected to the computer.

ota_update_task

This is the same git submodule that we used for Factory application earlier.

Each time the device powers up, it checks to see if an new interface app is available. If so, it will download the image and switches to the new version. [doc] To trigger an OTA update, power cycle the device, or press the EN button.

When the new interface app has critical errors, a rollback mechanism keep the device working by automatically rolling back to the previous working application.

After the app is downloaded using OTA, it is marked as pending_verify. Once the application determines that it is working well, it marks it as valid. On the other hand, when the application marks it as invalid or the device resets, the bootloader will boot the previous working application. [doc]

http server

Albeit not technically a task, we’ll discuss it here.

The HTTP server provides an UI to see the pool state and control a few settings.

The main_task uses the git module wifi_connect to establish a connection to the WiFi access point. Once established, it calls httpd_register_handlers to register the URI handlers for incoming GET requests.

These handlers are:

• httpd_root, replies to GET /
• httpd_ico“, replies to GET /favicon.ico
• httpd_json, replies to GET /json

httpd_root()

To save memory space on the ESP32, the device returns a minimum HTML page. The body part is replaced by the replace_body.js script.

httpd_ico()

The browser places the favicon next to the page’s title on the tab. This function simply sends an HTTP response containing the favicon.

httpd_json()

The Arduino replies to any command with the pool state information formatted as JSON. See illustration further down.

If query variables were passed, they will be send to the ipc->to_pool_q. The pool_task will create a corresponding message and send it to the pool controller.

The following query variables are supported:

        homeassistant/switch/opnpool/pool_circuit/set
        homeassistant/switch/opnpool/spa_circuit/set
        homeassistant/switch/opnpool/aux1_circuit/set
        homeassistant/switch/opnpool/aux2_circuit/set
        homeassistant/switch/opnpool/aux3_circuit/set
        homeassistant/switch/opnpool/ft1_circuit/set
        homeassistant/switch/opnpool/ft2_circuit/set
        homeassistant/switch/opnpool/ft3_circuit/set
        homeassistant/switch/opnpool/ft4_circuit/set
        homeassistant/climate/opnpool/pool_heater/set_mode
        homeassistant/climate/opnpool/pool_heater/set_tmp
        homeassistant/climate/opnpool/spa_heater/set_mode
        homeassistant/climate/opnpool/spa_heater/set_tmp
    

For the switches, it supports the values ON and OFF. For a thermostat (climate) mode, it supports the values off, heat, solar_pref and solar. For the thermostat temperature it accept a value in Fahrenheit.

Example

For details on web integration, refer to the Web UI section.

mqtt_task

IBM’s MQ telemetry transport (MQTT) is a lightweight, publish-subscribe network protocol that transports messages between devices. The keyword is telemetry, or remote monitoring.

MQTT is rapidly becoming the prevailed protocols to connect large numbers of devices. The mqtt_task implements an MQTT client.

Responsibilities of the task:

• Connect to MQTT broker.
• Initially, check if there is a coredump in flash memory, then publish it to MQTT.
• Subscribe to control topics, for diagnostic commands.
• Subscribing or publishing on MQTT topics on behalf of other processes.

Connect to MQTT broker

The code calls esp_mqtt_client_init and waits until the MQTT connection with the broker is established. It learns about a successful connection when it the callback function receives a MQTT_EVENT_CONNECTED.

If the connection were to be broken, the ESP IDF SDK will automatically reconnect to the broker.

Coredump to MQTT

As the devices that are deployed in the field, they have no serial monitor connected. Instead, we configure the ESP-IDF so the coredump is written to flash memory. Next time the ESP32 boots up, and it finds a coredump in flash, it uses the git submodule ESP32_coredump-to-server to publish the coredump to MQTT under the topic opnpool/data/coredump/ID.

Subscribe to control topics

Initially, the device asks MQTT task to subscribe to the topics listed below. Here ID is an unique device identifier.

        opnpool/ctrl
        opnpool/ctrl/ID
    

When another devices publishes a message on these topics, the MQTT task will handle the message. It supports the following values:

• who, request device info such as device id, firmware version, WiFi details, MQTT details and heap
• restart, restarts the device
• htstart, starts heap trace (when enabled)
• htstop, stops heap trace (when enabled)

When the MQTT task receives a message on these topics, it will take action and publish a reply to pool/data/ID/ as shown below.

Subscribe/publish on behalf of others

The hass_task queues a message to ipc->to_mqtt_q in order to subscribe or publish on a topic. The mqtt_task forwards this request to the ESP-IDF SDK.

When subscribing, the topic is also added to the _subscribers list, in case the connection with the broker needs to get re-established.

When another MQTT client publishes on a topic that another process subscribed to, the message is forwarded to the ipc->to_pool_q.

hass_task

Home Assistant is an open source home automation that puts local control and privacy first.

The hass_task’s duties are

• Listen for commands over MQTT.
• Support MQTT device discovery.

Listen for commands over MQTT

Initially, the device subscribes to the MQTT topics that lets other MQTT clients change pool controller settings such as thermostat set point or heating mode.

It subscribes to the topics listed below.

        homeassistant/switch/opnpool/pool_circuit/set
        homeassistant/switch/opnpool/spa_circuit/set
        homeassistant/switch/opnpool/aux1_circuit/set
        homeassistant/switch/opnpool/aux2_circuit/set
        homeassistant/switch/opnpool/aux3_circuit/set
        homeassistant/switch/opnpool/ft1_circuit/set
        homeassistant/switch/opnpool/ft2_circuit/set
        homeassistant/switch/opnpool/ft3_circuit/set
        homeassistant/switch/opnpool/ft4_circuit/set
        homeassistant/climate/opnpool/pool_heater/set_mode
        homeassistant/climate/opnpool/pool_heater/set_tmp
        homeassistant/climate/opnpool/spa_heater/set_mode
        homeassistant/climate/opnpool/spa_heater/set_tmp
    

When another devices publishes a message on these topics, the MQTT task will forward forward it as a network message to the Pool task. The latter will call hass_tx_pair to handle the message.

For the switches, it supports the values ON and OFF. For a thermostat (climate) mode, it supports the values off, heat, solar_pref and solar. For the thermostat temperature it accept a value in Fahrenheit.

MQTT device discovery

We auto populate the Home Assistant entities using MQTT Device Discovery. This enables us to use MQTT devices with only minimal configuration effort on the side of Home Assistant.

Every five minute, the device publishes MQTT discovery messages, to announce its presence to Home Assistant.

Some examples of these configuration topics are:

                topic = homeassistant/sensor/opnpool/air_temp/config;
                value = {
                    "~": "homeassistant/sensor/opnpool/air_temp",
                    "name": "Pool air temp",
                    "stat_t": "~/state",
                    "unit_of_meas": "°F"
                }
            

            topic = homeassistant/climate/opnpool/pool_heater/config;
            value = {
                "~": "homeassistant/climate/opnpool/pool_heater",
                "name": "Pool pool heater",
                "mode_cmd_t": "~/set_mode",
                "temp_cmd_t": "~/set_temp",
                "mode_stat_tpl": "{{ value_json.mode }}",
                "mode_stat_t": "~/state",
                "temp_stat_tpl": "{{ value_json.target_temp }}",
                "temp_stat_t": "~/state",
                "curr_temp_t": "~/state",
                "curr_temp_tpl": "{{ value_json.current_temp }}",
                "min_temp": 15,
                "max_temp": 110,
                "temp_step": 1,
                "temp_unit": "F",
                "avty_t": "~/available",
                "pl_avail": "online",
                "pl_not_avail": "offline",
                "modes": ["off","heat"]
            }
        

For details on Home Assistant integration, refer to the corresponding chapter.

From there the device publishes the data as e.g.

homeassistant/sensor/opnpool/air_temp/state 55
homeassistant/climate/opnpool/pool_heater/state {"mode":"off","target_temp":0,"current_temp":0}
homeassistant/climate/opnpool/pool_heater/available online

Rolling out a device to multiple sites introduces challenges. The WiFi credentials can no longer be part of the source code. Applying firmware updates or accessing crash information over USB becomes cumbersome. Even simply finding the IP address of the device on the LAN may be challenging.

To address such challenges we use a three tiered approach, in which the bootloader image decides what image to boot.

1. As usual, the bootloader image does some minimum initializations. If it finds a valid ota image, it passes control over to that image. If not, it starts the factory image.
2. The factory image takes care of provisioning WiFi and MQTT credentials with the help of a phone app. These credentials are stored in the nvs partition. It then downloads the ota image.
3. We refer to the ota image as the interface, as it provides the core of the functionality of the OPNpool device.

To facilitate this the flash memory is partitioned as shown below.

The partition table is shown in the table below.

The following sections describes the process in further detail. A whole chapter is dedicated to the interface image, which provides the core functionality of the OPNpool device.

Bootloader image

A few more words on the bootloader:

• If otadata partition is erased, it starts the factory image (assuming it is present).
• After the first OTA update, the ota_data partition is updated to specify which OTA app slot partition should be booted next.
• If the image works fine, it marks itself as ESP_OTA_IMG_VALID in the ota_data. Otherwise, it marks itself ESP_OTA_IMG_INVALID. This mechanism support image rollback to keep the device working after the update. It automatically rolls back to the previous version, if the image doesn’t pass its self test.

Factory image

The factory image, handles the provisioning and triggers an over-the-air (OTA) download of the interface image. It then reboots, the bootloader will start the interface image.

These steps are described in detail in the following section.

Provisioning WiFi credentials

For testing purposes, WIFI_CONNECT_SSID, WIFI_CONNECT_PASSWORD and OPNPOOL_MQTT_URL can also be provisioned using Kconfig. If empty, the device will use credentials from flash memory.

A phone is used to connect to the factory app using Bluetooth Low Energy (BLE). The factory image will advertise itself to the phone app. Using this phone the user specifies the WiFi and MQTT credentials. During the process the phone remains in contact with the ESP while the WiFi is set up and connects to an access point. The WiFi and MQTT credentials are stored in the nvs partition of flash memory.

You can find the source code of this app in the android directory. If you have an iOS phone, or you have problems running the Android app, you can extend esp_prov.py to include mqtt_url similar to what is shown here“.

The code base uses the git submodule ESP32_factory-ble-prov.

The figure and video below give an impression of the provisioning process

I (907) factory: Starting BLE provisioning
I (1338) ble_prov: advertising as "POOL_CC4504"
I (99768) ble_prov_handler: Received WiFi credentials:
        ssid Guest Barn
        password xxxxxxxxxx
I (100098) ble_prov_handler: WiFi Credentials Applied
I (100288) ble_prov_handler: Connecting ..
I (104328) factory: IP addr 10.1.1.118
I (104328) ble_prov: STA Got IP
I (104328) ota_task: Checking for OTA update (https://coertvonk.com/cvonk/pool/interface.bin)
I (104338) ota_task: Running from part "factory" (0x00010000)
I (104548) ota_task: Writing part ota_0 at offset 0x160000
I (104548) ota_task: Firmware on server: interface.f6d8367-dirty (Mar  2 2022 10:26:42)
I (104548) ota_task: Firmware running:   factory.6bc4c65-dirty (Mar  2 2022 14:52:42)
W (104558) ota_task: Downloading OTA update ..
I (105848) ble_prov_handler: Connected state
I (109668) ota_task: Wrote 5% of 1280 kB
:

Note to self: needs to check MQTT connection status and OTA download progress, awaiting answer to “Provisioning with custom-data after device establishes Wi-Fi connection“.

Connecting to the WiFi network

To establish the WiFi connection, the code base uses the git submodule ESP32_wifi-connect. This component makes this API available as a reusable component. It connects to a WiFi Access Point, and automatically reconnects when the connection drops.

Download the Interface image

The ESP-IDF API provides a mechanism that allows a device to update itself based on data received while the normal firmware is running. Our code base uses the git submodule ESP32_ota-update-task that makes the API available as a component.

The location of the factory image is specified by OTA_UPDATE_FIRMWARE_URL in Kconfig.

The code checks for an OTA update on a network server. If the image is different, it will download it. Upon completion, the device resets to activate the downloaded code. Note that we use the term “update” loosely, because it can also be used to downgrade the firmware.

To determine if the currently running code is different as the code on the server, it compares the project name, version, date and time. Note that these are not always updated by the SDK. The best way to make sure they are updated is by committing your code to Git and building the project from scratch (by removing the build directory).

During debugging, you may want to flash the image image directly as part of the “IDF-SDK: Build, flash and start monitor your device” cycle. To prevent that image from being overwritten by an OTA update, the update mechanism is disabled when the interface image is loaded in the factory partition.

An example of the update process is shown in the figure below.

I (3222) ota_task: Checking for OTA update (https://coertvonk.com/cvonk/pool/interface.bin)
    I (3232) ota_task: Running from part "factory" (0x00010000)
    I (3412) ota_task: Writing part ota_0 at offset 0x160000
I (3422) ota_task: Firmware on server: interface.f6d8367-dirty (Mar  2 2022 10:26:42)
I (3422) ota_task: Firmware running:   interface.6bc4c65 (Mar  2 2022 10:29:53)
W (3432) ota_task: Downloading OTA update ..
I (8142) ota_task: Wrote 5% of 1280 kB
:
I (23602) ota_task: Connection closed
I (24322) ota_task: Prepare to restart system!

More details of this component can be found at Github.

Describes the protocol that the pool controller uses on the RS-485 bus to communicate with its attached devices.

Key characteristics of this bus are:

• A pool controller initiates a query addressed to a specific device. The device will respond by performing an action and then respond back to the controller.
• It is an half-duplex connection, so only one system can transmit at the time. The controller will send a query and wait for a response, or times out when the device does not respond within a predefined period.

The OPNpool device sniffs this RS-485 bus to gather state information, and presents it to the user. The user can also change select parameters such as the thermostats or start e.g. the pool circuit.

The pool controller communicates with the pump and chlorinator over the RS-485 bus using a proprietary protocol. This protocol has been reverse engineered and documented in bits and pieces found on-line by people such as shown in the table below

The controller uses two different protocols to communicate with its devices:

• “A5” messages are used to communicate with devices such as pumps. The first bytes are always 0x00, 0xFF, 0xA5
• “IC” messages are used to communicate with the chlorinator. The first bytes are always 0x10, 0x02

A5 messages

The messages consist of a header, data and a 16-bit checksum. The length of the data block is specified in the header. The checksum is calculated starting at the last byte of the header (0xA5) up to the last data byte. Checksum is transmitted most significant byte first.

#define PACK8  __attribute__((aligned( __align__( uint8_t ) ), packed ))
typedef struct {
    uint8_t pro;  // protocol version
    uint8_t dst;  // destination
    uint8_t src;  // source
    uint8_t typ;  // message type
    uint8_t len;  // # of data bytes following
} PACK8 mHdr_a5_t;

The most significant 4 bits of source and destination address form the address group. The least significant 4 bits are the device number.

typedef enum {
    ADDRGROUP_all = 0x00,
    ADDRGROUP_ctrl = 0x01,
    ADDRGROUP_remote = 0x02,
    ADDRGROUP_chlor = 0x05,
    ADDRGROUP_pump = 0x06
} addrGroup_t;

We further distinguish between messages to/from a pump and control messages. Examples of control messages are time, state, head and scheduling requests.

Periodic state message from controller

Message on the RS-485 bus

Decoded message

datalink_pre = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
                0xFF, 0xFF, 0x00, 0xFF, 0xA5};

datalink_hdr_a5 = {
    .ver = 0x01,
    .dst = 0x0F,  // DATALINK_ADDRGROUP_ALL
    .src = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .typ = 0x02,  // NETWORK_TYP_CTRL_STATE_BCAST
    .len = 0x1D
};

network_msg_ctrl_state_bcast = {
    .hour       = 0x12,  // "18:26"
    .minute     = 0x1A,  //
    .activeLo   = 0x20,  // NETWORK_CIRCUIT_POOL | ..
    .activeHi   = 0x00   //
    .U04to06[3] = {0x00, 0x00, 0x00} // opt. more `active` circuits
    .U07to08[2] = {0x00, 0x00};
    .mode       = 0x00,
    .heatStatus = 0x00,  // false, false
    .U11        = 0x00,
    .delay      = 0x00,  // 0
    .U13        = 0x01,
    .poolTemp   = 0x34,  // 52 F
    .spaTemp    = 0x34,  // 52 F
    .U16        = 0x02,
    .solarTemp  = 0x50,  //
    .airTemp    = 0x37,  // 55 F
    .solarTemp2 = 0xFF,  // no-sensor
    .U20to21[2] = {0x02, 0xD0},
    .heatSrc    = 0x00,  // NETWORK_HEAT_SRC_OFF, NETWORK_HEAT_SRC_OFF 
    .U23to28[6] = {0x01, 0x00, 0x00, 0x00, 0x29 0x0B}
}

datalink_tail_a5 = {
    .crc[2] = {0x04, 0x28}  // 0x428
};

Controller requests status from pump

Message on the RS-485 bus

Decoded message

datalink_pre = {0xFF, 0x00, 0xFF, 0xA5};

datalink_hdr_a5 = {
    .ver = 0x00,
    .dst = 0x60,  // DATALINK_ADDRGROUP_PUMP
    .src = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .typ = 0x07,  // NETWORK_TYP_PUMP_STATUS
    .len = 0x00
};

network_msg_pump_status_req = {};

datalink_tail_a5 = {
    .crc[2] = {0x01, 0x1C};  // 0x11C
};

Pump replies with the message

Decoded message

datalink_pre = {0xFF, 0x00, 0xFF, 0xA5};

datalink_hdr_a5 = {
    .ver = 0x00,
    .dst = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .src = 0x60,  // DATALINK_ADDRGROUP_PUMP
    .typ = 0x07,  // NETWORK_TYP_PUMP_STATUS
    .len = 0x0F
};

network_msg_pump_status_resp = {
    .running  = 0x0A,  // true
    .mode     = 0x00,  // NETWORK_PUMP_MODE_FILTER
    .status   = 0x00,  // ok
    .powerHi  = 0x02,  // 748 W
    .powerLo  = 0xEC,  //
    .rpmHi    = 0x09,  // 2500 rpm
    .rpmLo    = 0xC4,  //
    .gpm      = 0x00,  // unknown G/m
    .pct      = 0x00,  // 0 %
    .U09      = 0x00,
    .err      = 0x00,  // no-error
    .remHr    = 0x00,
    .remMin   = 0x01,  // 1
    .clockHr  = 0x12,  // 18:26
    .clockMin = 0x1A   //
}

datalink_tail_a5 = {
    .crc[2] = {0x03, 0x1D}  // 0x31D
};

Controller requests status from chemical

Message on the RS-485 bus

datalink_pre = {0xFF, 0x00, 0xFF 0xA5};

datalink_hdr_a5 = {
    .ver = 0x00,
    .dst = 0x90,  // DATALINK_ADDRGROUP_X09
    .src = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .typ = 0xD2,  // NETWORK_TYP_CTRL_??
    .len = 0x01
};

unknown = {
    .U00 = 0xD2
}

datalink_tail_a5 = {
    .crc[2] = {0x02, 0xEA}  // 0x2EA
};

IC messages

The IC messages consist of a header, data and a tail containing a 8-bit checksum and a 2 byte post-amble. The length of the data block depends on the message type specified in the header. The checksum is calculated starting at the beginning of the header up to the last data byte.

#define PACK8  __attribute__((aligned( __alignof__( uint8_t ) ), packed ))
typedef struct {
    uint8_t dst;  // destination
    uint8_t typ;  // message type
} PACK8 mHdr_ic_t;

Controller asks chlorinator to set percentage

Message on the RS-485 bus

Decoded message

datalink_pre = {0x10, 0x02};

datalink_hdr_ic = {
    .dst = 0x50, // DATALINK_ADDRGROUP_CHLOR
    .typ = 0x11  // NETWORK_TYP_CHLOR_LEVEL_SET
};

network_msg_chlor_level_set {
    .pct = 0x50  // 80%
}

datalink_tail_ic = {
    .crc[1]  = {0xC3},
    .post[2] = {0x10, 0x03}
}

The chlorinator replies with

Decoded message

datalink_pre = {0x10, 0x02};

datalink_hdr_ic = {
    .dst = 0x00,  // DATALINK_ADDRGROUP_ALL
    .typ = 0x12   // NETWORK_TYP_CHLOR_LEVEL_RESP
};

network_msg_chlor_level_resp {
    .salt = 0x00,   // 0 ppm
    .err  = 0xC0    // POOLSTATE_CHLOR_STATUS_OK
}

datalink_tail_ic = {
    .crc[1]  = {0xE4},
    .post[2] = {0x10, 0x03}
}

Controller asks chlorinator for model number

Message on the RS-485 bus

Decoded message

datalink_pre = {0x10, 0x02};

datalink_hdr_ic = {
    .dst = 0x50,  // DATALINK_ADDRGROUP_CHLOR
    .typ = 0x14   // NETWORK_TYP_CHLOR_X14
};

unknown = {
    .U00 = 0x00
}

datalink_tail_ic = {
    .crc[1]  = {},
    .post[2] = {0x10, 0x03}
}

The chlorinator replies

Decoded message

datalink_pre = {0x10, 0x02};

datalink_hdr_ic = {
    .dst = 0x00,  // DATALINK_ADDRGROUP_ALL
    .typ = 0x03   // NETWORK_TYP_CHLOR_NAME
};

network_msg_chlor_name = {
    .U00 = 0x00,
    .name[16] = {0x49, 0x6E, 0x74, 0x65,  // Intellichlor--40
                    0x6C, 0x6C, 0x69, 0x63,
                    0x68, 0x6C, 0x6F, 0x72,
                    0x2D, 0x2D, 0x34, 0x30}
}

datalink_tail_ic = {
    .crc[1]  = {0xBC},
    .post[2] = {0x10, 0x03}
}

Controller pings the chlorinator

Message on the RS-485 bus

Decoded message

datalink_pre = {0x10, 0x02};

datalink_hdr_ic = {
    .dst = 0x50,  // DATALINK_ADDRGROUP_CHLOR
    .typ = 0x00   // NETWORK_TYP_CHLOR_PING_REQ
};

network_msg_chlor_ping_req = {
    .U00 = 0x00
}

datalink_tail_ic = {
    .crc[1]  = {0x62},
    .post[2] = {0x10, 0x03}
}

A5 information requests

Request heat settings

Messages on the RS-485 bus

datalink_pre = {0xFF, 0x00, 0xFF, 0xA5};

datalink_hdr_a5 = {
    .ver = 0x01,
    .dst = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .src = 0x20,  // DATALINK_ADDRGROUP_REMOTE
    .typ = 0xC8,  // NETWORK_TYP_CTRL_HEAT_REQ
    .len = 0x00
};

network_msg_ctrl_heat_req = {}

datalink_tail_a5 = {
    .crc[2] = {0x01, 0x9E}  // 0x19E
};

Controller replies with

datalink_pre = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
                0xFF, 0xFF, 0x00, 0xFF, 0xA5};

datalink_hdr_a5 = {
    .ver = 0x01,
    .dst = 0x0F,  // DATALINK_ADDRGROUP_ALL
    .src = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .typ = 0x08,  // NETWORK_TYP_CTRL_HEAT
    .len = 0x0D
};

network_msg_ctrl_heat = {
    .poolTemp         = 0x34,  // 52 F
    .spaTemp          = 0x34,  // 52 F
    .airTemp          = 0x37,  // 55 F
    .poolTempSetpoint = 0x04,  // 4 F
    .spaTempSetpoint  = 0x00,  // 0 F
    .heatSrc          = 0x00,  // NETWORK_HEAT_SRC_OFF
    .U06to12[7]       = {0x00, 0x00, 0xFF, 0x00, 0x00, 0x00, 0x00
}

datalink_tail_a5 = {
    .crc[2] = {0x02, 0x7C}  // 0x27C
};

Request schedule

Messages on the RS-485 bus

Remote requests

datalink_pre = {0xFF, 0x00, 0xFF, 0xA5};

datalink_hdr_a5 = {
    .ver = 0x01,
    .dst = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .src = 0x20,  // DATALINK_ADDRGROUP_REMOTE
    .typ = 0xDE,  // NETWORK_TYP_CTRL_SCHED_REQ
    .len = 0x00
};

network_msg_ctrl_sched_req = {}

datalink_tail_a5 = {
    .crc[2] = {0x01, 0xB4}  // 0x1B4
};

Controller replies with

datalink_pre = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
                0xFF, 0xFF, 0x00, 0xFF, 0xA5};

datalink_hdr_a5 = {
    .ver = 0x10,
    .dst = 0x0F,  // DATALINK_ADDRGROUP_ALL
    .src = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .typ = 0x1E,  // NETWORK_TYP_CTRL_SCHED
    .len = 0x10
};

network_msg_ctrl_sched_resp {
    .U00to03[4] = {0x01, 0x00, 0x00, 0x0F},
    .sched[2] = {{
        .circuit    = 0x00,  // NETWORK_CIRCUIT_SPA
        .U05        = 0x00,
        .prgStartHi = 0x00,
        .prgStartLo = 0x00,
        .prgStopHi  = 0x00,
        .prgStopLo  = 0x00,
    },{
        .circuit    = 0x06,  // NETWORK_CIRCUIT_POOL
        .U05        = 0x01,
        .prgStartHi = 0x01,  // 08:00
        .prgStartLo = 0xE0,
        .prgStopHi  = 0x02,  // 10:00
        .prgStopLo  = 0x58,
    }
}

datalink_tail_a5 = {
    .crc[2] = {0x02, 0x45}  // 0x245
};

Request current time

Messages on the RS-485 bus

Remote requests

datalink_pre = {0xFF, 0x00, 0xFF, 0xA5};

datalink_hdr_a5 = {
    .ver = 0x01,
    .dst = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .src = 0x20,  // DATALINK_ADDRGROUP_REMOTE
    .typ = 0xC5,  // NETWORK_TYP_CTRL_TIME_REQ
    .len = 0x00
};

network_msg_ctrl_time_req = {}

datalink_tail_a5 = {
    .crc[2] = {0x01, 0x9B}  // 0x19B
};

Controller replies with

datalink_pre = {};

datalink_hdr_a5 = {
    .ver = 0x01,
    .dst = 0x0F,  // DATALINK_ADDRGROUP_ALL
    .src = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .typ = 0x05,  // NETWORK_TYP_CTRL_TIME
    .len = 0x08
};

network_msg_ctrl_time {
    .hour     = 0x12,  // 19:26
    .minute   = 0x1A,
    .U02      = 0x10,
    .day      = 0x18,  // 2022-02-24
    .month    = 0x02,
    .year     = 0x16,
    .clkSpeed = 0x00,
    .daylightSavings = 0x00
}

datalink_tail_a5 = {
    .crc[2] = {0x01, 0x3E}  // 0x13E
};

A5 set requests

Set circuit

Message on the RS-485 bus

datalink_pre = {0xFF, 0x00, 0xFF, 0xA5};

datalink_hdr_a5 = {
    .ver = 0x01,
    .dst = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .src = 0x22,  // DATALINK_ADDRGROUP_REMOTE (wireless_remote)
    .typ = 0x86,  // NETWORK_TYP_CTRL_CIRCUIT_SET
    .len = 0x02
};

network_msg_ctrl_circuit_set = {
    .circuit = 0x06,  // 1-based circuit
    .value = 0x01
};

datalink_tail_a5 = {
    .crc[2] = {0x01, 0x67}  // 0x167
};

Controller replies with

datalink_pre = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
                0xFF, 0xFF, 0x00, 0xFF, 0xA5};

datalink_hdr_a5 = {
    .ver = 0x01,
    .dst = 0x22,  // DATALINK_ADDRGROUP_REMOTE (wireless_remote)
    .src = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .typ = 0x01,  // NETWORK_TYP_CTRL_SET_ACK
    .len = 0x01
};

network_msg_ctrl_set_ack = {
    .typ = 0x86;  // NETWORK_TYP_CTRL_CIRCUIT_SET
};

datalink_tail_a5 = {
    .crc[2] = {0x01, 0x60}  // 0x160
};

Set heat

Message on the RS-485 bus

datalink_pre = {0xFF, 0x00, 0xFF, 0xA5};

datalink_hdr_a5 = {
    .ver = 0x01,
    .dst = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .src = 0x22,  // DATALINK_ADDRGROUP_REMOTE (wireless_remote)
    .typ = 0x88,  // NETWORK_TYP_CTRL_HEAT_SET
    .len = 0x04
};

struct network_msg_ctrl_heat_set_t {
    .poolSetpoint = 0x48,  // 0
    .spaSetpoint  = 0x16,  // 1
    .heatSrc      = 0x01,  // 2
    .UNKNOWN_3    = 0x00   // 3
};

datalink_tail_a5 = {
    .crc[2] = {0x01, 0xC3}  // 0x1C3
};

Controller replies with

datalink_pre = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
                0xFF, 0xFF, 0x00, 0xFF, 0xA5};

datalink_hdr_a5 = {
    .ver = 0x01,
    .dst = 0x22,  // DATALINK_ADDRGROUP_REMOTE (wireless_remote)
    .src = 0x10,  // DATALINK_ADDRGROUP_CTRL
    .typ = 0x01,  // NETWORK_TYP_CTRL_SET_ACK
    .len = 0x01
};

network_msg_ctrl_set_ack = {
    .typ = 0x88;  // NETWORK_TYP_CTRL_HEAT_SET
}

datalink_tail_a5 = {
    .crc[2] = {0x01, 0x62}  // 0x162
};

Describes methods for debugging the pool interface. Includes JTAG debugging and post-mortem coredump analysis.

Run time debug

Using `menuconfig`, the user can configure different levels to console logging. The verbosity is specified as None (0), Errors (1), Warnings and errors (2).

• OPNPOOL_DBGLVL_DATALINK
• OPNPOOL_DBGLVL_NETWORK
• OPNPOOL_DBGLVL_POOLSTATE
• OPNPOOL_DBGLVL_POOLTASK
• OPNPOOL_DBGLVL_MQTTTASK
• OPNPOOL_DBGLVL_HASSTASK
• OPNPOOL_DBGLVL_HTTPD

Post mortem

The device includes support to generate core dumps on unrecoverable errors. This allows post-mortem analysis of the software state at the moment of failure. This way you can find what task, the instruction and what call stack lead to the crash. The core dump is written to the coredump partition in flash.

As we will see in the Interface chapter, the mqtt_task will read this coredump from flash and forward it over MQTT.

GDB over JTAG

The ESP32 supports the Open On-Chip Debugger. Say goodbye to ESP_LOG, and explore the world that was once the exclusive domain of in-circuit emulators. Using special pins on the ESP32, your computer can set break points, inspect variables and single step instructions.

ESP32 board and JTAG adapter

The JTAG interface hooks directly to the CPU. That allows it do stop the CPU, set breakpoints and have access to whatever the CPU has access to.

The JTAG header is available on the OPNpool board. This 10 pin header should be connected to a JTAG/USB interface, such as the ESP-PROG. With the interface, make sure to set the JTAG PWR SEL jumper for 3.3V. Also try to keep the connection short (<10 cm).

OpenOCD and driver

Under Windows, install the 64-bit FTDI D2xx Driver setup executable;

1. Using micro-USB cables, connect JTAG adapter and the serial port on the OPNpool board to the computer.
2. Wait until USB ports are recognized by Windows and drivers are installed. If they do not install automatically, then do so manually using the driver setup from FTDI.
3. Install and run the Zadig tool (≥ v2.7).
   • Options » List All Devices
   • The list of devices should contain two ESP-PROG specific USB entries with driver name “FTDIBUS (vxxxx)”:
     • Dual RS232-HS (Interface 0), connected to the JTAG port
     • Dual RS232-HS (Interface 1), connected to the UART port
   • Select Dual RS232-HS (Interface 0), and reinstall attached driver to the “WinUSB (v6xxxxx)”, see the picture below.
4. Windows Device Manager should no longer show the JTAG port, because WinUSB is a user-space driver.

Compile and debug

Set up the OpenOCD configuration

1. F1 » Select OpenOCD Board Configuration.
2. Choose the ESP32 chip (via ESP-PROG). (Unless you use the ESP-WROVER-KIT of course.)
   • Note that WROVER modules use a 1.8 SPI flash because of PSRAM limitations.
   • Others such as WROOM modules use 3.3V SPI flash. This flash voltage is important, because GPIO12 is shared with SPI flash. Selecting the wrong voltage may cause flash uploads to fail.

See if OpenOCD starts

1. F1 » ESP-IDF: OpenOCD Manager.
2. Choose Start OpenOCD.

Build the code with symbols and without optimalizations.

1. F1 » ESP-IDF: Launch GUI configuration tool
2. Specify the Debug (-Og) compiler optimalization level.
3. Press ^e d to build, upload and monitor over the serial port.
   • If the “search documentation” keyboard shortcut is also assigned to ^e d, then first remove that binding.
   • Note that you can also upload the binary over JTAG (program_esp filename.bin 0x10000 verify).

From the VSCode debug side bar (^e d), click on the green arrow at the top and select Launch to connect to the OPNpool device.

Notes

• There are only two hardware breakpoints
• After hitting a break point, you may still have to select the corresponding task
• To determine the image location in flash, OpenOCD uses the address of the first `app` in the partition table. When using OTA updates, this will be the factory image instead of the OTA downloaded application. To work around this, specify esp32 appimage_offset (see docs.espressif.com).
• You may only want to debug the 1st core (set ESP32_ONLYCPU 1)
• Espressif has more info in Espressif JTAG Debugging

Describes the development environment for the OPNpool interface as described in VSCode-starters ESP32, consisting of

• GNU toolchain
• Espressif IoT Development Framework (ESP-IDF)
• Microsoft Visual Code Studio IDE
• Git version control

Git

We use git with a repository hosted on GitHub.

1. Make sure that you can ssh to github.com. On Windows, you may have to install the Windows OpenSSH Client.
2. Clone the repository and its submodules to a local directory. Note the --recursive flag.

   git clone --recursive git@github.com:cvonk/OPNpool.git

EAGLE

The schematics and layout of the printed circuit board were created using Autodesk EAGLE. If you wish to edit these files, you will need to install the software.

Remember

In Windows, you need to specify the EAGLE_HOME environment variable that points to the your EAGLE directory. Inside EAGLE it is known as $HOME and is used in Control Panel » Options » Directories.

The schematic and board layouts are available in the hardware directory. Custom parts such as the Wemos LOLIN-D32, DC/DC converter, JTAG, RS-485 connector can be found in the hardware/libraries/OPNpool.lbr library.

Go to Options then Directories, and add the absolute path of `OPNpool/hardware/libraries` to the libraries.

ESP Toolchain

These instructions are for Windows, the paths will be slightly different on other operating systems.

Before you start, make sure that Git is installed and that you can call git from the command line.

If you haven’t already, install Microsoft Visual Studio Code (VSCode).

From within VSCode:

1. Add the Microsoft’s C/C++ extension
2. Add the Espressif IDF 1.4.0 extension.
3. Install the tools and configure the Espressif IDF extension.
   • F1 » ISP-IDF: Configure ESP-IDF
   • Select Advanced mode
   • ESP-IDF 4.4 in C:/espressif (goes to subdir esp-idf)
   • GNU Tools in C:/espressif/bin
   • Choose “Download ESP-IDF Tools”
4. To speed up compile times, disable Windows Defender’s real-time scanning of C:/espressif. If you get an “You’ll need a new app to open this WindowsDefender link” prompt, then issue Get-AppxPackage Microsoft.SecHealthUI -AllUsers | Reset-AppxPackage from an elevated PowerShell terminal.

If you prefer to use a ESP-IDF beta version or the master branch, refer to VSCode starter for ESP32.

Start humble

1. From Visual Studio Code, change to an empty folder and copy a simple example project such as blink.
   • F1 » ESP-IDF: Show ESP-IDF Example Projects
     • Get-started » Blink
     • Click on “Create project using example Blink”.
2. Connect an ESP32 board to your computer. It will show up as a Serial Port device in the Windows device manager.
3. Select the serial port in Visual Studio Code
   • F1 » ESP-IDF: Device configuration
     • Device Target = ESP32
     • Device Port = COM4
     • Baud Rate = 115200
4. Start a compile-flash-monitor cycle
   1. F1 » ESP-IDF: Build, flash and monitor, or use ^e d.
   2. Flash using the “UART”.

In case you encounter warnings or errors, refer to the table below.

Compiling

You should now be able to compile and link e.g. the “factory” application.

Start Visual Studio Code (vscode), in the directory OPNpool\factory.

1. Connect an ESP32 board to your computer. In Device Manager, the UART (COM) will show up under the Serial Ports. Select this port in Visual Studio Code:
   • F1 » ESP-IDF: Device configuration
     • Device Target = ESP32
     • Device Port = COM6
     • Baud Rate = 115200
2. Start a compile-flash-monitor cycle
   1. F1 » ESP-IDF: Build, flash and monitor, or using ^e d.
   2. Flash using the “UART”.

Android Studio

The OPNpool provisioning application is available on the Play Store. The Deploying chapter of this document describes the process in more detail.

If you prefer to build it yourself, download and install Google Android Studio and its SDK. We’d recommend trying to build one of the example projects before you building the OPNpool provisioning app.

Build

Open Android Studio in the android directory and start the build process. Enable Developer mode on you Android device, and enable Wireless Debugging. Your phone should now be listed as a test device. Click the button to run the app on your phone.

Note that the app targets API 30 (Android 11), but builds with API 31. This is so that we can request BLUETOOTH_SCAN and BLUETOOTH_CONNECT permissions for Android 12 devices. For details refer to the Android 12 Behavior Changes document

The design of OPNpool hardware is based around a RS-485 Transceiver and an ESP32 System on a Chip (SoC). The transceiver converts between the RS-485 differential signals and the transmit/receive signals used by the UART in the SoC. The ESP32 is supported by the well documented Espressif IoT Development Framework (ESP-IDF).

Early prototypes

We started by using a single threaded task on a Espressif ESP8266 SoC. This was cheap and offered built-in WiFi, but over time, we migrated to the newer ESP32. Though still very affordable, and it also offers Bluetooth LE and runs FreeRTOS which helps in separating the software components.

ESP32 SoC

The official name of the ESP32 SoC is “ESP32-D0WDQ6”. It contains two CPUs and peripherals such as WiFi, Bluetooth and Bluetooth LE.

The ESP32-WROOM32 module combines this ESP32 with a clock circuit, 4 MByte flash memory and a trace antenna. This in turn is mounted on a Wemos LOLIN D32 daughterboard, that provides the reset logic, USB bridge and battery circuit. The main reason for using the LOLIN D32, is that we didn’t want to solder the micro-USB connector ourselves. Looking back, it seems more cost effective as well.

Schematic

The schematic itself is pretty straightforward. We take the power from the pool controller and convert it to 3.3 V, and a buck converter provides 5 Volts to the battery connector on the ESP32 daughterboard. Using the battery input helps prevent problems when the circuit is also powered through the USB connector

The data path is between the RS-485 connector and the ESP32 on the LOLIN D32 daughterboard. There is an optional terminator resistor to prevent reflections on the bus. The JTAG header is for debugging as detailed in the Debugging chapter.

The second revision adds LEDs for RS-485 RX (green) and TX (amber) to simplify setup. It also introduces a push button to preform a factory reset, and increases the bore hole size for the screws.

Bill of Materials

The price comes down to under $40 based on single quantity, excluding shipping and taxes. However, note that some items, such as the PCB, have minimum order quantities.

Layout

The schematic fits easily on a two layer PCB. Note the cut out for the RF antenna.

Behind the scenes

Just for fun, a little stop animation video of the PCB design and assembly