Making HTTP Requests Over MQTT

HTTP and MQTT are two protocols that may seem at complete odds with each other, but both are suitable for IoT applications. I created what is essentially a 50-line Python script that enables you to make an HTTP request via the action of publishing an MQTT message. Why would anyone want to do this? There may be instances where you have an IoT device with an HTTP/REST API but its IP address is either unknown or private, and this would enable you to interface with it through a common MQTT broker.

Design

This script is meant to be run as a service on the IoT device(s) whose IP/whereabouts may be unknown. It works by having the devices connect to a remote MQTT broker and subscribe to a topic specific to each device. The topic string will contain an HTTP verb and path, and the payload, if present, can contain query parameters, headers, and request body. Once a message is received the script uses the topic and payload to make an HTTP request to the configured API and return the response over a similar MQTT topic.

The script depends on requests for making HTTP requests, and paho-mqtt for MQTT.

API

HTTP paths and MQTT topics are both slash delimited so the conversion works out well. The device's request topic looks like {app_name}/req/{id}/+/# where the + wildcard represents the HTTP verb and the # wildcard represents the (optional) path of the endpoint. The request payload can be left empty, or it can contain a JSON object with optional values:

{
    "params": {{PARAMS}}, # Flat dictionary of URL Query Parameters
    "body": {{BODY}}, # Body either JSON or a string
    "headers": {{HEADERS}}, # Flat dictionary of case-insensetive headers
    "req_id": {{REQUEST_ID}} # Value to be sent back with response; no impact on API call
}

Once the MQTT message is received, the corresponding HTTP request is made and the response is sent via an MQTT message whose topic is identical to the request topic but req is replaced with res. The response message payload is JSON that will always contain these values:

{
    "content": {{CONTENT}}, # Either JSON or string if XML, HTML, TXT, etc.
    "status": {{HTTP_STATUS_CODE}}, # Integer, e.g. 200, 404, etc.
    "req_id": {{REQ_ID}} # Same value from request if provided, or null
}

Configuration

Configuration is done through environment variables.

The use case is that the device is connected to a common, remote MQTT broker, so MQTT_HOST should actually be something like mqtt.example.com, not localhost, and the MQTT username and password should also be set; the broker should certainly not allow anonymous connections for any serious use. Conversely, the HTTP_BASE_URL default value of http://localhost/ is actually sensible as the API would be running on the device, likely unencrypted and bound to localhost. Basic auth is supported, and the credentials are set on the device, because passing credentials over MQTT with every request would be incredibly boneheaded. MQTT_TOPIC_START and MQTT_TOPIC_ID correspond to {app_name} and {id} from the API section above. These values can have slashes; leading and trailing slashes will be trimmed, so ///iot/config/v1//// will be interpreted as iot/config/v1.

Callback

After grabbing configuration from the environment and connecting to the MQTT broker, everything happens in a single callback function that runs in a new thread to avoid blocking other queued requests. requests is smart enough to handle None parameters and set default headers.

def _on_message(client, userdata, msg):
    topic_arr = msg.topic.split('/', PATH_INDEX)
    verb = topic_arr[VERB_INDEX]
    path = topic_arr[PATH_INDEX] if len(topic_arr) > PATH_INDEX else ''
    
    try:
        payload = json.loads(msg.payload)
    except ValueError:
        payload = None

    body_json = body_data = params = headers = req_id = None

    if type(payload) is dict:
        params = payload.get('params')
        body = payload.get('body')
        headers = payload.get('headers')
        req_id = payload.get('req_id')
        if type(body) is dict or type(body) is list:
            body_json = body
        else:
            body_data = body
        if type(headers) is not dict: headers = None

    r = requests.request(method=verb, url=HTTP_BASE_URL+path, params=params, data=body_data, json=body_json, auth=HTTP_AUTH)
    print(r.status_code, r.request.url)
    try:
        content = r.json()
    except ValueError:
        content = r.content.decode('utf-8')

    client.publish(topic='/'.join([MQTT_PUB_TOPIC,verb,path]).rstrip('/'), payload=json.dumps({'content': content, 'status': r.status_code, 'req_id': req_id}), qos=2)
    

Examples

Say we have a device with a JSON REST API running on port 8000 with a some endpoints that allow us to read GPS coordinates. Given an {app_name} of myapp/v1, a unique device {id} of L0L69, and the device's base URL of http://localhost:8000/rest/api/v1/ the device would subscribe to the remote broker on topic myapp/v1/req/L0L69/+/#. When a message arrives on topic myapp/v1/req/L0L69/get/gps/latitude with an empty payload, the script wll take the verb get and path gps/latitude and make the call GET http://localhost:8000/rest/api/v1/gps/latitude. Once the API call returns, the script will publish a message on topic myapp/v1/res/L0L69/get/gps/latitude with payload

{
    "content": {
        "latitude":38.88888
    },
    "status": 200,
    "req_id": null
}

That worked, and we didn't even need to provide a payload since it was a GET.

Building off of the previous example, let's say our device was a smart fridge all along with a food related API. A POST may look something like this: A request message would arrive on topic myapp/v1/req/L0L69/post/food/dessert with payload

{
    "body": {
        "dessertId": "applePie",
        "type": "pie",
        "notes": [
            "good with ice cream",
            "quintessentially American"
        ]
    },
    "req_id": "request_1337"
}

The script would make the call POST http://localhost:8000/rest/api/v1/food/dessert with the body from the payload, and the API's response would prompt an MQTT message to be published on topic myapp/v1/res/L0L69/post/food/dessert with payload

{
    "content": "Yum. Resource created at /rest/api/v1/food/dessert/applePie",
    "status": 201,
    "req_id": "request_1337"
}

For some reason our fictional RESTful JSON API responded with TXT content, but the script handles non-JSON just fine.

Considerations

While the scheme of HTTP is syncronous - a client makes a request and awaits the response - MQTT is a asyncronous, as there may be any number of messages on any number topics being published at any given time. For this reason, the application publishing the request messages should wait until it successfully subscribes to the response message topic before making any requests to avoid race conditions. This is also the reason for including the optioanl "req_id" key in the request and response payloads.

The HttpOverMqtt script provides an MQTT API for making HTTP calls. I found this http to mqtt bridge, which appears to do the inverse of what this application does; it sends an MQTT message upon making an HTTP POST. Something like this can be used or adapted to create an HTTP API that interfaces with HttpOverMqtt, allowing you to make HTTP requests to devices whose IPs may be unknown or unreachable using MQTT as a hidden middle layer.

Another use for HttpOverMqtt would be as a really terrible VPN. You could set HTTP_BASE_URL to a remote URL and run the script on a remote machine.

Security

If you're actually going to use this for something important, take extreme care to ensure your MQTT broker's ACL is configured properly so that not everyone connected to the broker can listen in and make their own requests at will. You should also use MQTTS, which uses SSL/TLS, so that your requests are encrypted. MQTT_PORT defaults to 8883 (mqtts) for this reason. The MQTT_TOPIC_ID must be unique for each device, so a device would only be listening to requests destined for itself, and not the other one-thousand smart-fridges connected to the broker. Assuming the MQTT broker being used is Mosquitto, there are a few ways to handle authentication. Mosquitto supports auth plugins for authenticating in via SQL, HTTP, Redis, etc. If you're not using an auth plugin, you can use mosquitto's password_file and acl_file. If you can trust each device to keep to its own dev-specific topic, you can use one password entry with a user we'll call device. The ACL entry for that user would look like this:

user device
topic read {app_name}/req/+/+/#
topic write {app_name}/res/+/+/#
# First '+' wildcard represents MQTT_TOPIC_ID (no slashes)
# If MQTT_TOPIC_ID contains slashes, you'll need more '+' wildcards for each

If you cannot trust each device to keep to its own topic, you can create a username and password for each device. The username, MQTT_USER, should be the same as MQTT_TOPIC_ID in this case. The ACL would pattern entries, which apply to all users:

pattern read {app_name}/req/%u/+/#
pattern write {app_name}/res/%u/+/#
# The username is substituted for '%u', restricting each device to its own topic

The user making the requests, let's say admin, should have ample ACL permissions to write to the req topic and read the res topic for all device names. If you have other users that are using the MQTT broker for other applications but don't want them to be able to access the HttpOverMqtt set of topics, I have a pull request for the official Mosquitto repository that would allow you to deny topics explicitly in the ACL that would otherwise be allowed by a more permissive ACL entry. It would be used like this:

user otheruser
topic readwrite #
topic deny {app_name}/#
# Access any topic except for {app_name}/#

While readwrite # gives access to all topics, the deny {app_name}/# would deny access to topics matching {app_name}/# even though readwrite # would normally grant it. If you don't have this option from my mosquitto fork, you will have to give explicit access to every topic that the other users would use:

user otheruser
topic readwrite hello/#
topic readwrite world/#
topic readwrite foo/#
topic readwrite bar/#

A final option, which I haven't implemented yet, would be use to SSH encryption to authenticate the requests. The backend making the requests would need to know the public SSH keys of all the devices beforehand. The application creates an MQTT request and encrypts the payload using the device's public SSH key. Upon receipt, the device would decrypt the payload using its private SSH key and make the request. The response would be encrypted using the private SSH key, and the application receiving it would decrypt using the public SSH key.

Conclusion

Thanks for reading. I'd be interested to hear about anyone's use of this project. Writing this post took me longer than it did to write the actual script. HttpOverMqtt respository can be found here .