Nallely, a hackable organic brain for signal experimentation

Nallely (pronounced "Nayeli") is not a tool. It's not even an instrument in the usual sense. Nallely is a living system you can enter, perturb, and observe. It's a programmable environment where signals exist, interact, mutate, and sometimes surprise you. Nallely connects your MIDI gear (software, hardware) with virtual signal processor to create a whole instrument while keeping each device running independently. As consequence, things interact naturally and can drift in a non-deterministic way. Patch an LFO to your synths' filter, stack a sequencer, cross-modulate everything with feedback loops everywhere if you want and let it evolve. Nallely runs perfectly on a Raspberry Pi and is tested daily on a rpi5, rpi3, and rpi-zero2w together (distributed computation context).

You can think of Nallely as a small brain. Each device is a neuron (nothing related to AI): it receives signals, transforms them, and emits signals. Neurons are autonomous, asynchronous, and fully patchable. You can connect them in ways that would never happen in a sober brain, closer to what might emerge under psychedelic influences, creating unusual signal flows that generate unexpected behaviors.

Some neurons abstract the physical world. MIDI devices become voices: synths that produce sound, controllers that emit gestures. Other neurons act as senses: a webcam module can become an eye, feeding statistics or motion into the system. Visuals can be thought of as mental imagery, shaped by how signals propagate through the network. Future audio modules will give the system the ability to "hear" and react to sound.

At its core, Nallely is about signal shaping. Every patch is a question: how do I want to shape this signal?

Signals can be split, merged, quantized, integrated, transformed, delayed, folded, or fed back into themselves. Neurons range from pitch shifters, harmonizers, sequencer to integrators and Laplace transformations. There is no strict separation between control, modulation, or data: everything is a signal, and everything can be patched. More importantly, you can create your own neurons in Python while the system runs, you don't have to stop your system to code an idea, give birth to it while your living system runs, no restart needed. You can also create your neuron in any technology as long as it speaks websocket.

You can design playable instruments, auto-playable systems, or things that resist both categories. Make it useful. Make it useless. Interact with it, listen to it, watch it evolve, or let it run and observe what emerges.

Connect anything to anything. See what happens.

What Nallely Is Not

Nallely is not a DAW, and it is not a system for computer-based sound synthesis. It is a meta-synth and a companion to MIDI instruments: a way to combine, bend, and rewire existing synths into new instruments, or to make semi-modular setups more modular.

If you’re looking for something that ensures strong strict time, Nallely is not for you. If you’re looking for high-predictability, Nallely might not be for you.

Asynchrony is a first-class feature. There is no global clock, no enforced synchronization. Timing emerges because things interact, not because they are aligned.

Nallely is not a DAW. It is not a DSP engine. It is a playground for emergent behaviors, generative control, and turning synths into living systems, programmable, perturbable, and sometimes surprising.

Nallely's Website Demo videos (new and old)

Control multiple MIDI devices	Patch your devices	Monitor the signals	Change settings

Manage your patchs as a versioned memory	Trevor is always here	Get a Smalltalk-like playground	Explore your patch in 3D

Features

• modular-synthesis like for CV signals that can be output as MIDI note, MIDI CC, MIDI pitchweel value, ...,
• abstraction to access your MIDI devices,
• virtual devices (LFO, Harmonizer, Sequencer, Quantizer, ...) you can connect to your MIDI devices, as source to control your MIDI device or as target to be controled by your MIDI controller/synth, generate patterns/sequences, use/write reactive devices to alter what you are playing or how you play it,
• MIDI bridge to drive any software or VST/LADSPA that supports MIDI (headless or not). Tested with VCV Rack, AMSynth, Yoshimi, Hexter, Bristol, ZynAddSubFx, FluidSynth.
• each device is hosted as an independent thread: each device has its own life cycle and are operating at real wall clock time (it's reality time, not audio real-time),
• jitter mitigation,
• web interface relying on a websocket protocol (named Trevor) which allows you to do graphically what you would ask Nallely to do programmatically (create devices, map them, change parameters, alter scalers),
• interactive code playground in the browser (through Trevor UI) inspired by Smalltalk playground,
• small web-based widget integrated in the web interface (scopes 1D to 3D, pads, XYpads, gameboy emulator),
• save/reload preset for any MIDI device at run time,
• save/reload/manage memory slots: each memory slot contains a snapshot of a Nallely session which is versionned (Nallely memory is now saved as a git repository!),
• bind/unbind control/pad/key of your MIDI devices between each other or virtual devices, converting the CC between source and target if required,
• bind/unbind the velocity of the pad/key of your MIDI devices to any CC control,
• bind/unbind pad/key individualy to any control, note, parameter of MIDI devices or virtual devices,
• bind/unbind a key/pad to another one (even if not the same note, you can map a note to its octave on the same device or another one),
• scaler for the values that goes from a source to a target: you can restrict the range of values that will be sent to the target,
• auto-scaling: the source device adapts its output range to the target without setting the range yourself,
• introspective API for auto-adaptive virtual modules: virtual device can impact the running system including connections through the whole system meaning that you can achieve "self-modifying patch",
• programmatic seemless API to your MIDI Device,
• links are formally defined and are entities of the domain,
• bouncy links: links can trigger target port associated link to have reaction chains,
• Python API code generator for your device if it is listed by the MIDI CC & NRPN database project, or from a YAML description you can easily write (see configs folder for examples),
• easy API to build your own reactive virtual devices (neurons) in Python, or as external virtual device in any technology,
• websocket-based bus on which external services can auto-register and expose parameters to which you can bind your MIDI/virtual devices in a seemless way,
• possibility to send/receive/broadcast/listen messages and information from the external services,
• LFOs composition with mathematical expressions,
• random preset generator for MIDI devices and virtual devices,
• full random patch generator (basic at the moment) with auto-generative capacity as virtual device (you can control it from MIDI devices or other virtual devices),
• create Python devices directly from the web UI:
  • while the system runs and plays: modify, create, debug using pdb/ipdb, or whatever python debugger you have installed on your machine,
  • Smalltalk/lisp-like hot-patch: instances are not recreated, they are migrated to a new class,
  • isolated hot-patch you can patch only a single instance of a class without disturbing the others,
  • full git-based versionning for each individual device,
  • debug a single instance of a device class without impacting the others,
  • exception handler mechanism: if a modification fails, only the impacted instances is stopped, you can then manually fix it and restart it.
• Distribute computation accross multiple sessions on the local network
  • Scan network for friend sessions
  • Expose virtual neurons to friend sessions in a transparent way
  • Distribute computation accross the local network
• (currently disabled) bind/unbind any Python function to any control/pad/key of your MIDI Device,

Planned:

Browse the Issues section of the repository to see what's planned.

Quick Installation & Quickstart

You can install Nallely from source or use the precompiled binary for your system. The installation from source is detailed lower in the README.

Install from binaries

This repository proposes pre-compiled binaries that can directly be downloaded from the Release section in this repository. Those binaries are built from the repository by Github action. The binaries are produced by pyinstaller and embedd Nallely and Trevor, that can be ran from the command line, as well as Trevor-UI and visuals that can be accessed directly from your local web-browser.

1. Download the archive related to your OS. Please, note that the binary for Windows has been built, but not tested.
2. Unzip the archive, you should have a single binary file that contains Nallely with Trevor.
3. Run Nallely: nallely run --with-trevor --serve-ui (this will run Nallely, enables the Trevor protocol and serve the UI as a webapp). 3bis. If you want to have the builtins MIDI devices API loaded add the option -b: nallely run --with-trevor --serve-ui -b. Currently the builtin devices are the Korg NTS-1 and Korg Minilogue. There is extra configuration in configs for the Roland S-1, Behringer JT-4000 micro, and Behringer ProVS mini.

Once you have Nallely running, you should see a prompt. Pressing enter will display information about the running Nallely's session. Typing q shutsdown the session, ? displays the help menu. As Nallely have been run with Trevor and serves the UI, you can directly go to http://localhost:3000 or http://127.0.0.1:3000 which will serve Trevor-UI. The UI should connect directly to the Trevor Websocket Server. Check the quick tour/doc of Trevor-UI

Connect a MIDI Device

When it comes to the communication and the connection with MIDI devices, Nallely let you the choice about how you'll interact with it:

• from a generated Python API,
• from a MIDI bridge (generic Python API).

The main difference is in the way you'll patch the elements later, both solution are obviously valid, but one gives you a better representation of the features of your MIDI device.

• Generated Python API implies the generation of the Python API from a higher-level description, either CSV or YAML, and to generate the API from Nallely directly using the command line,
• MIDI bridge doesn't imply the generation of the Python API, but will provide also a less detailed meaningful way of patching your devices: you'll have to rely on your knowledge of CCs of your MIDI device.

The generated Python API lets you describe properly your MIDI device and had meaningful names to each CC, while the MIDI bridge exposes CCs ports on which you'll connect to.

Control your MIDI Device using the MIDI bridge (without Generating a Dedicated API)

In the MIDI device rack (on the left in vertical mode), choose the MIDI Bridge from the drop-down menu. You'll have a new MIDI neuron created.

Expand the MIDI IOs top panel, you'll see the MIDI ports discovered by the system. You can then click on the port that hosts your MIDI device, then click on the MIDI Bridge instance to connect them.

Once it's done, you're good to go, you can then select a channel for your device by clicking on the device and expanding the right bar, then you can start to patch!

The following video shows how to connect the MIDI Bridge to a MIDI port, change the channel and create an LFO to connect to the notes of the MIDI Bridge.

output.mp4

Include a new MIDI device Generating a Dedicated API

If you have your MIDI device listed in the MIDI CC & NRPN database as CSV, or if you have a YAML description of your MIDI device, you can generate the Python API to integrate it with Nallely using the generate subcommand:

1. download the CSV configuration for your MIDI device (e.g: Korg NTS-1)
2. generate the YAML configuration and Python API: nallely generate -i YOURFILE.csv -o YOURFILE.py
3. once you have the Python API for your MIDI device, just include it on the command line with the -l option: nallely run --with-trevor --serve-ui -l PATH/TO/YOURFILE.py

If you have already a YAML description, the command to generate the Python API is the same: nallely generate -i YOURFILE.yaml -o YOURFILE.py.

Having a key section

By default, if you generated the Python API from the CSV, the generated Python API will not have a "keys" section (referencing the keys and notes of your synth), you'll need to add it manually:

1. open the generated YAML file with a text editor
2. add a notes: 'keys_or_pads' entry in it, as well as a pitchwheel: 'pitchwheel' in a section an existing section, or create a new one (for example, in the Roland S-1 configuration)
3. re-generate the new Python API using the YAML file as input: nallely generate -i YOURFILE.yaml -o YOURFILE.py
4. include it in the command line as before using the -l option.

NOTE: The notes: 'keys_or_pads' entry doesn't have to be in an isolated section, it can be set with other sections, but it's only possible to have one key_or_pads entry by section.

Embedded Visuals

All the embedded visuals are available at http://localhost:3000/visuals. You can open them, and to manipulate them from Nallely and Trevor-UI, you need first to create a WebsocketBus in Nallely. You can do that by opening the dropdown menu of the middle vertical rack in the UI and clicking on WebsocketBus (currently we are limited to 1 bus, but many different visuals can register to a same bus).

Architecture

Nallely and Trevor is based on a simple classical decoupled architecture:

• Nallely defines a kind of small object model (devices, parameters, links between parameters and their semantic) relying on threads (1 instance of device by thread), and message sending between them (signal sent from a device parameter to another),
• Trevor defines the protocol to manipulate Nallely "from the outside", and exposes this API programmatically and through a websocket server.
• Trevor-UI defines a web-UI that communicates with a running Nallely's session through the Trevor Websocket Server.

The following diagram shows a kind of instance of what a Nallely's session can look like.

flowchart BT
  TrevorUI["Trevor UI (web)"]

  subgraph TrevorWSS["Trevor WebSocket Server"]
    direction TB
    Nallely["Nallely Session"]
    subgraph Nallely["Nallely Session"]
      direction TB
      MIDI["MIDI Device API"] --> Virtual2["EG"]
      MIDI2["MIDI Device 2 API"]
      MIDI2 --> Virtual2
      MIDI2 --> Virtual1
      Virtual2 --> MIDI2
      Virtual1["LFO"] --> Virtual2
      Virtual1 --> MIDI
      Virtual1 --> MIDI2
      WSBus["Websocket Bus"]
      LFO2 --> WSBus
      MIDI2 --> WSBus
    end
    TrevorProto["Trevor Protocol"]
    Nallely <--> TrevorProto
  end

  WSBus --> WSBPort["⭘ WebSocket Bus Port"]
  TrevorUI -.-> Port
  MIDIDevice["MIDI Device"] <--> MIDI
  MIDIDevice2["MIDI Device 2"] <--> MIDI2
  WSBPort --> Visual1
  WSBPort --> Visual2
  WSBPort --> Oscilloscope1
  TrevorProto --> Port["⭘ Trevor WebSocket Port"]

Loading

A Nallely's session is made of a set of Virtual Devices instances (LFOs, EG, Harmonizer, etc), and MIDI Device instances that connects to physical MIDI devices through a dedicated API. The API can be written manually or generated from a YAML decription. In a session, virtual devices and MIDI devices exposes parameters which can all be patched. Consequently, you can have a LFO that controls the speed of another one, and the output of this modulated LFO can modulate another one. All devices inside a Nallely session are patchable: you can also patch parameter inside a same device. Consequently, you can have a parameter of your MIDI device that contols multiple parameters of other MIDI devices, or even parameter of the same MIDI device (e.g: the cutoff that controls also the resonance in an inverted fashion, or on a dedicated small range only). A websocket bus, also a virtual device, hence patchable, allows visuals and external application to register to it, exposing their ports. Once their ports are exposed on the bus, any MIDI or virtual device inside the Nallely's session can connect to them and control them.

If you want more details about the architecture, you can check this link which gives a good more or less in depth explanation of all the parts of the architecture. Not everything is 100% exact, but it's really close from reality.

Documentation

A first version of the UI documentation gives you a tour of the main features of Nallely and Trevor-UI, and how to start your first patches in a visual way.

A first draft about how Nallely can help you declare your devices and map them using the current API can be find in the documentation. The documentation has not been updated with the latest capabilities of the API, but it should be retrocompatible, unless I really missed something.

Going Lower Level

Scripting and various options in Nallely

### Trevor, Nallely's companion

Trevor is a communication protocol and a UI made to communicate with Nallely through websocket and ask Nallely to create device instance, map devices together or apply scaler. Trevor proposes a web UI that lets you bind everything at run time, without any need for stopping/starting again scripts, as well as an interactive code playground inspired by Smalltalk playgrounds that let's you code/script on the fly, or even create your own module/neuron from the interface without having to restart the system. Trevor runs in two parts: the websocket server (the backend), and the frontend. The frontend is merly just a representation of the state given by the backend, it holds no important state and always render what the backend sends. This means that Trevor-UI is a web interface, but any UI that connects to this websocket, renders the received state and sends commands that complies with the Trevor API (to be documented), it's possible to have another type of UI. There is no limit, Trevor-UI is a simple UI, but other UIs focusing on new interactions and other representation will be developed in the future.

Local Dev Installation

In dev mode, or pure local mode, you can run Trevor-UI this way: the web UI is built with vite, react, and uses yarn. We consider here that you have all of this installed already. To install Trevor:

cd trevor
yarn install

Then to launch everything:

# in 1 terminal, inside of the "trevor" directory
yarn dev

# in another terminal, there is various other options you can pass, try --help to see all of them
nallely run --with-trevor

Old Screenshots/videos of Trevor UI I want to see them, so much memories...

output.1.mp4

Screenshots of Trevor UI (really old)

Launch the example (run Nallely in headless scripting mode)

This repo comes with one example of a spiral that is controlled by LFOs created by Nallely. To launch it, once you have installed Nallely or downloaded the

There is 4 ways of launching the demo:

1. either using the Nallely binary you can download from the release section,
2. launching the visual-spiral.py script if you have Python installed on your machine and Nallely installed as library (pip installed),
3. loading the visual-spiral.nly patch through the UI,
4. loading by a script the visual-spiral.nly patch.

We will demonstrate here the three first ways: using the visual-spiral.py script, using the downloaded binary, and using the UI.

Using the downloaded binary

Once you have the binary download, you can run any python script that relies on Nallely's internal API without the need of Trevor. This is handy when you need to have simple scripts, but not to load Trevor, and without using the UI.

1. Simply copy those files from this repository:

• visual-spiral.py => core system for this small example, creates 2 LFOs, waits for external modules (spiral and possibly terminal-based oscilloscope) to connect and maps all together,
• spiral.hml => simple three.js spiral controlled by some parameters,

2. Launch nallely run -i visual-spiral.py,
3. Open spiral.html in your browser,
4. ...
5. Profit

Using the Python script

This way of doing relies on the assumption that you have Python installed on your machine and Nallely installed through pip install (see section below)

1. Simply copy those file from this repository:
   • visual-spiral.py => core system for this small example, creates 2 LFOs, waits for external modules (spiral and possibly terminal-based oscilloscope) to connect and maps all together,
   • spiral.hml => simple three.js spiral controlled by some parameters,
   • external_scope.py => simple terminal-based oscilloscope relying on plotext (optional).
2. Launch python -m http.server 8000 in the project repository and go with your browser to http://localhost:8000/spiral.html,
3. Launch visual-spiral.py
4. ...
5. Profit
6. (Optional) if you want to see the LFO shape, launch external-scope.py from another terminal.

The screenshot below shows you what the result looks like with everything launched

Using Trevor UI

visual-load-smaller.mp4

Other quick examples using Nallely's scripting API

Here is a simple example about how to map the cutoff of the KORG NTS-1 with the cutoff of the KORG Minilogue, in an inverse fashion:

import nallely
from nallely.devices import NTS1, Minilogue

nts1 = NTS1()
minilogue = Minilogue()

try:
  nts1.filter.cutoff = minilogue.filter.cutoff.scale(127, 0)

  input("Press enter to stop...")
finally:
  nallely.stop_all_connected_devices()

Another example is how to bind the velocity of a pad of the Akai MPD32 to the cutoff of the Minilogue:

import nallely
from nallely.devices import MPD32, Minilogue

mpd32 = MPD32()
minilogue = Minilogue()

try:
  minilogue.filter.cutoff = mpd32.pads[36].velocity

  input("Press enter to stop...")
finally:
  nallely.stop_all_connected_devices()

Another more complex example where we create a simple harmonizer for the Minilogue, where the NTS-1 is also playing the harmonized note:

import nallely
from nallely.devices import NTS1, Minilogue

scale = [0, 2, 2, 1, 2, 2, 2]  # major scale
intervals = [4, 3, 3, 4, 4, 3, 3]  # 3rd intervals
nts1 = NTS1()
minilogue = Minilogue()

try:
  for root_note in range(0, 127, 12):  # We start on lower C key and iterate on each octaves
    note = root_note
    for config in zip(scale, intervals):
        offset, interval = config
        note += offset  # we compute the next note of the scale from the root
        new_note = note + interval  # we add the corresponding interval
        if new_note > 127:  # if the result goes over 127, no need to map
            break

        # here is the important part
        minilogue.keys[new_note] = minilogue.keys[note]  # we map the key to the 3rd on the minilogue
        nts1.keys[new_note] = minilogue.keys[note]  #  we map the key to the 3rd on the NTS-1
  input("Press enter to stop...")
finally:
  nallely.stop_all_connected_devices()

This is one way of creating a harmonizer, there is a better way that is already embedded in Nallely as a pre-existing neuron.

How to install from sources

See instructions

The current version requires Python >= 3.10. The library relies mainly on `mido` and `python-rtmidi`, so your system needs to support them.

Local Installation from sources

There is currently no pypi package for it, so the easiest way to install the library is to:

1. create a virtual env
2. pip install git+https://github.com/dr-schlange/nallely-midi.git

Usage

Nallely's scripts are pure Python script and can be launched on their own, but to simplify the integration with Trevor (protocol/UI), or to avoid to have to write the try/finally/input, or to generate the Python API from a CSV or a YAML file configuration, there is a convenient command line interface.

$ nallely -h
usage: nallely [-h] {run,generate} ...

Playground for MIDI instruments that let's you focus on your device, not the exchanged MIDI messages

positional arguments:
  {run,generate}
    run           Run scripts and Trevor (protocol for remote control)
    generate      Generate a Python API for a MIDI device

options:
  -h, --help      show this help message and exit

Run a script or Trevor

The command line let's you either run a simple script (an "init scrip"), launch Trevor websocket server to connect later Trevor's UI, to include paths where you might have API for devices, or include the builtins device config (Korg NTS-1, Korg Minilogue).

$ nallely run -h
usage: nallely run [-h] [-l [LIBS ...]] [--with-trevor] [--serve-ui] [-b] [--experimental] [-i INIT_SCRIPT]

options:
  -h, --help            show this help message and exit
  -l, --libs [LIBS ...]
                        Includes one or more paths (file or directory) where to look for MIDI devices API (includes those paths to Python's lib paths).
                        The current working directory is always added, even if this option is not used. The paths that are Python files will be
                        automatically imported.
  --with-trevor         Launches the Trevor protocol/websocket server
  --serve-ui            Serves Trevor-UI, and makes it accessible from your browser. This option is only activated if '--with-trevor' is used.
  -b, --builtin-devices
                        Loads builtin MIDI devices (Korg NTS1, Korg Minilogue)
  --experimental        Loads experimental virtuals devices
  -i, --init INIT_SCRIPT
                        Path towards an init script to launch. If used with "--with-trevor", the script will be launched *before* Trevor is started.

Generates a Python API for a MIDI device from a CSV or YAML configuration

If you have your MIDI device listed in this repository as CSV, or a YAML description of your MIDI device, you can generate the Python API for it to be integrable with Nallely.

$ nallely generate -h
usage: nallely generate [-h] -i INPUT -o OUTPUT

options:
  -h, --help           show this help message and exit
  -i, --input INPUT    Path to input CSV or YAML file
  -o, --output OUTPUT  Path to the file that will be generated

NOTE: If a CSV configuration is given as input, the equivalent YAML configuration will be generated at the same time. Also, as you can see, there is no special mention of key/pads section in the CSV configuration available in the https://github.com/pencilresearch/midi repository. If you want to generate a key section for your device, you can modify the YAML configuration by adding a xxx: 'keys_or_pads' entry. Here is how it's done for the Korg NTS-1:

KORG:
  NTS-1:
    # ... Other sections here
    keys:
      notes: 'keys_or_pads'

The xxx: 'keys_or_pads' entry doesn't have to be in an isolated section, it can be set with other sections, but it's only possible to have one key_or_pads entry by section.

Install Nallely and Trevor on a Raspberry Pi

Nallely and Trevor works perfectly fine on raspberry pi, meaning that you can bring them with you, connect all your synth to the RPI and enjoy your organic meta-synth system everywhere with you. The configuration is currently still very manual (systemd service creation, etc), but will be made automatic later.

TO WRITE

Fonts used

Nallely includes several retro Amiga-style fonts from the rewtnull/amigafonts project.

These fonts are licensed under the GNU GPL with the Font Exception, which allows them to be used and embedded in this BSD-licensed project without imposing copyleft requirements.

See trevor/public/fonts/LICENSE.amigafonts for full license and attributions.