README.md

AI Questionaire Backend

The AI questionaire backend is an iExec application responsible for taking the input from the user (currently as a textual prompt in the "email" field of the protectedData, due to historical reasons, passing it to the AI model (currently, a tiny version of Qwen2.5 embedded into the image itself), and finally returning it as a result using iExec's result mechanism.

Unfortunatelly, despite all our attempts to get that working with iExec, the only thing we could go working is a mock node.js version, which can be found in ./node-mocked. Everything else failed in one way or another, and this part of the repository is here to document the various ways in which we couldn't get it to work.

Architecture

flowchart
    subgraph User
        UserUser[User]
        UserUser ---|Survey Answers| Prompt
        ResultList
    end
    subgraph iExec
        subgraph DataProtectorCore
            Prompt ---|ProtectData| PromptDataset
        end
        UserUser ---|ProcessData| Backend
        subgraph Backend
            PromptDataset --> PromptFile
            PromptFile --> Exec
            Model --> Exec
            Exec --> Results
             Results ---|ProcessDataResult| ResultList
        end
    end

Loading

List of attempts

#	Attempt name	Description	Status
1	python-sconify	An attempt to follow iExec's Build your first application with Scone framework documentation.	❌ DCAP validation fail on both Prod and Debug workerpools.
2	python-transformers	An attempt to use the Huggingface transformers library for the AI model.	⭕ Abandonded as we experiment with simpler Python SGX examples first
3	python-gramine	Attempt to use Gramine instead of Scone following the iExec's unlisted Build a Gramine application documentation.	❌ iExec does not support Gramine at the moment
4	node-mocked	The default hello World" app generated by idapp-cli	✔️ Deployed as 0x05F88328fAe2Ac1271C68f2E65864692c3AD9B0A (but does no AI)
5	nodejs-transformers.js	An attempt to use the transformers.js library by Huggingface.	❌ SCONE's Node.js version (14) is too old to support transformers.js.
6	nodejs-execFile	An attempt to use the llamafile executable with the Node version we have	❌ SCONE limitations: ENOSYS when attempting to spawn the process.
7	nodejs-wllama	An attempt to use the @wllama/wllama library (also based on llama.cpp) for inference.	❌ SCONE's Node.js version (14) doesn't support the WebAssembly exception handling extension wllama requires.
8	python-scone-py	An attempt to use the official Scone build image, as suggested by Scone's documentation.	❌ DCAP validation fail when running on Prod; maybe mrenclave mismatch on Debug.
9	python-sconify-debug	An attempt to pass --allow-debug-mode to sconify_iexec (after staring at code for way too long), with no observable result.	❌ DCAP validation fail on both prod and debug worker pools.
10	nodejs-llama.node	An attempt to use yet another library for invoking llama.cpp, @fugood/llama.node	⭕ Ran out of time to debug properly, needs an extra library and testing. Could conceivably be made work with sconify_iexec ... --dlopen=2, tho that's insecure

Note that if you want to try all of those for yourself, you would need an account on SCONE's registry whitelisted for use with iExec, as described in Build your first application with Scone framework, and might need to deploy/publish/run the app using commands like as described in 1, 2,:

iexe init && iexec app init && iexec order init # To generate iexec.json

# With any of the non-node examples:

#update iexec.json to include the pushed image in the app section
iexec app deploy

# With any of the node examples:
#update deployed.json to include appid
#update iexec.json to include appid in the order section
iexec order sign --app && iexec order publish --app

# To finally run:
iexec app run --tag tee,scone --workerpool debug-v8-bellecour.main.pools.iexec.eth --watch

NOTE: The code in this folder was scraped off a messy attempts folder on a best-effort basis, and as such comes with NO WARRANTY of any kind, including FITNESS for any particular purpose, including reproducing specific bugs.

General observations

In general, all of the attempts involved trying to run either Python or Node.js.

All the Python attempts ended up hitting the same problem related to DCAP validation, which had something to do with having a SCONE policy not allowing debug-mode SGX enclaves. Since the built/sconified image doesn't seem to contain a policy at all, and the error is the same on both the Prod and Debug pools, and the Node.js image being perfectly happy to run in SGX debug mode, we are not at all sure what's going on.

Meanwhile, all of the Nodejs attempts used the idapp-cli API server provided and hosted by iExec, since it seemed to be successful in building at least the mock Node.js app. However, as SCONE doesn't have prebuilt images for Node >14, all the newer libraries ended up throwing errors related to newer JavaScript feature, like ??=, or to missing runtime APIs, like Blob or parts of WebAssembly.

Overall, would say that iExec's reliance on SCONE ended up frustrating virtually all of our attempts to run an AI model inside iExec's platform. Unfortunately, Gramine support had been axed a while ago, so we couldn't try using its virtualization model to run our applications. Better yet, actual Intel TDX/AMD SEV-SNP support, such as that afforded by e.g. Constellation, (WIP) CoCo, or (shameless plug, WIP) Apocryph, would have meant that we would have been able to run any code, without needing finicky conversions to get it inside Intel SGX.

Future work

A few more things we could have tried:

• Get node.llama working all the way.
• Run a C++ project that includes llama.cpp inside SCONE, as SCONE seems better suited to running compiled languages than interpreted languages.
• Investigate if there is something else we could have tried for python-sconified -- including different versions of the sconification software that might have worked in the past.