Yehowshua/ZIm
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Remove non-WASI code

Browse source
This commit is contained in:
Isaac Clayton 2022-06-06 11:00:05 +02:00
parent 805c86b781
commit 38d7321511
23 changed files with 211 additions and 712 deletions
Download patch file Download diff file Expand all files Collapse all files

4
crates/plugin_runtime/Cargo.toml
View file

@ -4,10 +4,8 @@ version = "0.1.0"
edition = "2021"
[dependencies]
mlua = { version = "0.8.0-beta.5", features = ["lua54", "vendored", "serialize"] }
serde = "1.0"
map-macro = "0.2"
wasmtime = "0.37.0"
wasmtime-wasi = "0.37.0"
anyhow = { version = "1.0", features = ["std"] }
serde = "1.0"
bincode = "1.3"

48
crates/plugin_runtime/README.md
View file

@ -1,6 +1,48 @@
# Zed's Plugin Runner
This crate contains a fairly generic interface through which plugins may be added to extend the editor. Currently the intention of this plugin runtime is language server definitions.
Wasm plugins can be run through `wasmtime`, with supported for sandboxed system integration through WASI. There are three `plugin` crates that implement different things:
Anything that implements the `Runtime` trait may be used as a plugin. Plugin interfaces are declared by implementing the `Interface` trait.
1. `plugin_runtime` loads and runs compiled `Wasm` plugins, and handles setting up system bindings.
Wasm plugins can be run through `wasmtime`. We plan to add wasi support eventually. We also plan to add macros to generate bindings between Rust plugins compiled to Wasm and the host runtime.
2. `plugin` is the crate that Rust Wasm plugins should depend on. It re-exports some required crates (e.g. `serde`, `bincode`) and provides some necessary macros for generating bindings that `plugin_runtime` can hook into.
3. `plugin_macros` implements the proc macros required by `plugin`, like the `#[bind]` attribute macro.
## ABI
The interface between the host Rust runtime ('Runtime') and plugins implemented in Wasm ('Plugin') is pretty simple.
`Buffer` is a pair of two 4-byte (`u32`) fields:
```
struct Buffer {
ptr: u32,
len: u32,
}
```
All functions that Plugin exports must have the following properties:
- Have the signature `fn(ptr: u32, len: u32) -> u32`, where the return type is a pointer to a `Buffer`, and the arguments are the component parts of a `Buffer`.
- The input `Buffer` will contain the input arguments serialized to `bincode`.
- The output `Buffer` will contain the output arguments serialized to `bincode`.
- Have a name starting with two underscores.
Additionally, Plugin must export an:
- `__alloc_buffer` function that, given a `u32` length, returns a `u32` pointer to a buffer of that length.
Note that all of these requirements are automatically fullfilled for any Rust Wasm plugin that uses the `plugin` crate, and imports the `prelude`.
Here's an example Rust Wasm plugin that doubles the value of every float in a `Vec<f64>` passed into it:
```rust
use plugin::prelude::*;
#[bind]
pub fn double(mut x: Vec<f64>) -> Vec<f64> {
x.into_iter().map(|x| x * 2.0).collect()
}
```
All the serialization code is automatically generated by `#[bind]`.

23
crates/plugin_runtime/build.rs
View file

@ -1,23 +0,0 @@
use std::process::Command;
fn main() {
let cwd = std::env::current_dir().unwrap();
let plugin_workspace = cwd.join("plugin").join("Cargo.toml");
Command::new("cargo")
.args(&["clean", "--manifest-path"])
.arg(&plugin_workspace)
.status()
.unwrap();
Command::new("cargo")
.args(&[
"build",
"--release",
"--target",
"wasm32-unknown-unknown",
"--manifest-path",
])
.arg(&plugin_workspace)
.status()
.unwrap();
println!("cargo:warning=recompiling plugins")
}

0
crates/plugin_runtime/heck.txt
View file

82
crates/plugin_runtime/plugin/Cargo.lock generated
View file

@ -1,82 +0,0 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 3
[[package]]
name = "bincode"
version = "1.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1f45e9417d87227c7a56d22e471c6206462cba514c7590c09aff4cf6d1ddcad"
dependencies = [
"serde",
]
[[package]]
name = "cargo_test"
version = "0.1.0"
dependencies = [
"bincode",
"plugin",
"serde",
]
[[package]]
name = "plugin"
version = "0.1.0"
dependencies = [
"bincode",
"plugin_macros",
"serde",
]
[[package]]
name = "plugin_macros"
version = "0.1.0"
dependencies = [
"bincode",
"proc-macro2",
"quote",
"serde",
"syn",
]
[[package]]
name = "proc-macro2"
version = "1.0.39"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c54b25569025b7fc9651de43004ae593a75ad88543b17178aa5e1b9c4f15f56f"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quote"
version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a1feb54ed693b93a84e14094943b84b7c4eae204c512b7ccb95ab0c66d278ad1"
dependencies = [
"proc-macro2",
]
[[package]]
name = "serde"
version = "1.0.137"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61ea8d54c77f8315140a05f4c7237403bf38b72704d031543aa1d16abbf517d1"
[[package]]
name = "syn"
version = "1.0.96"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0748dd251e24453cb8717f0354206b91557e4ec8703673a4b30208f2abaf1ebf"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "unicode-ident"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d22af068fba1eb5edcb4aea19d382b2a3deb4c8f9d475c589b6ada9e0fd493ee"

2
crates/plugin_runtime/plugin/Cargo.toml
View file

@ -1,2 +0,0 @@
[workspace]
members = ["cargo_test"]

16
crates/plugin_runtime/plugin/cargo_test.lua
View file

@ -1,16 +0,0 @@
query = [[(
(attribute_item
(meta_item
(identifier) @test)) @attribute
.
(function_item
name: (identifier) @name) @funciton
)]]
function run_test(name)
local command = 'cargo test -- ' .. name
local openPop = assert(io.popen(command, 'r'))
local output = openPop:read('*all')
openPop:close()
return output
end

7
crates/plugin_runtime/plugin/cargo_test/Cargo.lock generated
View file

@ -1,7 +0,0 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 3
[[package]]
name = "cargo_test"
version = "0.1.0"

9
crates/plugin_runtime/plugin/cargo_test/Cargo.toml
View file

@ -1,9 +0,0 @@
[package]
name = "cargo_test"
version = "0.1.0"
edition = "2021"
[dependencies]
serde = "1.0"
bincode = "1.3"
plugin = { path = "../../../plugin" }

14
crates/plugin_runtime/plugin/cargo_test/src/main.rs
View file

@ -1,14 +0,0 @@
use plugin::prelude::*;
#[bind]
pub fn sum_lengths(strings: Vec<String>) -> usize {
let mut total = 0;
for string in strings {
total += string.len();
}
return total;
}
pub fn main() -> () {
()
}

14
crates/plugin_runtime/src/lib.rs
View file

@ -1,16 +1,2 @@
use mlua::{Function, Lua, LuaSerdeExt, Value};
use serde::{de::DeserializeOwned, Serialize};
pub use map_macro::{map, set};
pub mod runtime;
pub use runtime::*;
pub mod lua;
pub use lua::*;
pub mod wasm;
pub use wasm::*;
pub mod wasi;
pub use wasi::*;

62
crates/plugin_runtime/src/lua.rs
View file

@ -1,62 +0,0 @@
use mlua::Result;
use crate::*;
impl Runtime for Lua {
type Plugin = LuaPlugin;
type Error = mlua::Error;
fn init(module: Self::Plugin) -> Result<Self> {
let lua = Lua::new();
// for action in module.actions {
// action(&mut lua).ok()?;
// }
lua.load(&module.source).exec()?;
return Ok(lua);
}
// fn constant<T: DeserializeOwned>(&mut self, handle: &Handle) -> Result<T> {
// let val: Value = self.globals().get(handle.inner())?;
// Ok(self.from_value(val)?)
// }
fn call<A: Serialize, R: DeserializeOwned>(&mut self, handle: &str, arg: A) -> Result<R> {
let fun: Function = self.globals().get(handle.to_string())?;
let arg: Value = self.to_value(&arg)?;
let result = fun.call(arg)?;
Ok(self.from_value(result)?)
}
// fn register_handle<T: AsRef<str>>(&mut self, name: T) -> bool {
// self.globals()
// .contains_key(name.as_ref().to_string())
// .unwrap_or(false)
// }
}
pub struct LuaPlugin {
// name: String,
source: String,
// actions: Vec<Box<dyn FnOnce(&mut Lua) -> Result<(), ()>>>,
}
impl LuaPlugin {
pub fn new(
// name: String,
source: String,
) -> LuaPlugin {
LuaPlugin {
// name,
source,
// actions: Vec::new(),
}
}
// pub fn setup(mut self, action: fn(&mut Lua) -> Result<(), ()>) -> LuaPlugin {
// let action = Box::new(action);
// self.actions.push(action);
// self
// }
}

53
crates/plugin_runtime/src/main.rs
View file

@ -1,53 +0,0 @@
use mlua::Lua;
use plugin_runtime::*;
pub fn main() -> anyhow::Result<()> {
let plugin = WasmPlugin {
source_bytes: include_bytes!(
"../plugin/target/wasm32-unknown-unknown/release/cargo_test.wasm"
)
.to_vec(),
};
let mut sum = Wasm::init(plugin)?;
let strings = "I hope you have a nice day"
.split(" ")
.map(|x| x.to_string())
.collect();
let result = sum.sum_lengths(strings);
dbg!(result);
Ok(())
}
// struct SumLengths {
// sum_lengths: Handle,
// }
// impl Interface for SumLengths {
// fn from_runtime<T: Runtime>(runtime: &mut T) -> Option<Self> {
// Some(SumLengths {
// sum_lengths: runtime.handle_for("sum_lengths")?,
// })
// }
// }
// impl SumLengths {
// fn sum_lengths<T: Runtime>(&self, runtime: &mut T, strings: Vec<String>) -> Option<usize> {
// runtime.call(&self.sum_lengths, strings).ok()
// }
// }
// #[plugin::interface]
trait SumLengths {
fn sum_lengths(&mut self, strings: Vec<String>) -> usize;
}
impl<T: Runtime> SumLengths for T {
fn sum_lengths(&mut self, strings: Vec<String>) -> usize {
let result = self.call("sum_lengths", strings).ok().unwrap();
return result;
}
}

77
crates/plugin_runtime/src/runtime.rs
View file

@ -1,77 +0,0 @@
// use std::Error;
use serde::{de::DeserializeOwned, Serialize};
// /// Represents a handle to a constant or function in the Runtime.
// /// Should be constructed by calling [`Runtime::handle_for`].
// #[derive(Debug, Clone, Hash, PartialEq, Eq)]
// pub struct Handle(String);
// impl Handle {
// pub fn inner(&self) -> &str {
// &self.0
// }
// }
// /// Represents an interface that can be implemented by a plugin.
// pub trait Interface
// where
// Self: Sized,
// {
// /// Create an interface from a given runtime.
// /// All handles to be used by the interface should be registered and stored in `Self`.
// fn from_runtime<T: Runtime>(runtime: &mut T) -> Option<Self>;
// }
pub trait Runtime
where
Self: Sized,
{
/// Represents a plugin to be loaded by the runtime,
/// e.g. some source code + anything else needed to set up.
type Plugin;
/// The error type for this module.
/// Ideally should implement the [`std::err::Error`] trait.
type Error;
/// Initializes a plugin, returning a [`Runtime`] that can be queried.
/// Note that if you have any configuration,
fn init(plugin: Self::Plugin) -> Result<Self, Self::Error>;
// /// Returns a top-level constant from the module.
// /// This can be used to extract configuration information from the module, for example.
// /// Before calling this function, get a handle into the runtime using [`handle_for`].
// fn constant<T: DeserializeOwned>(&mut self, handle: &Handle) -> Result<T, Self::Error>;
/// Call a function defined in the module.
fn call<A: Serialize, R: DeserializeOwned>(
&mut self,
handle: &str,
arg: A,
) -> Result<R, Self::Error>;
// /// Registers a handle with the runtime.
// /// This is a mutable item if needed, but generally
// /// this should be an immutable operation.
// /// Returns whether the handle exists/was successfully registered.
// fn register_handle<T: AsRef<str>>(&mut self, name: T) -> bool;
// /// Returns the handle for a given name if the handle is defined.
// /// Will only return an error if there was an error while trying to register the handle.
// /// This function uses [`register_handle`], no need to implement this one.
// fn handle_for<T: AsRef<str>>(&mut self, name: T) -> Option<Handle> {
// if self.register_handle(&name) {
// Some(Handle(name.as_ref().to_string()))
// } else {
// None
// }
// }
// /// Creates the given interface from the current module.
// /// Returns [`Error`] if the provided plugin does not match the expected interface.
// /// Essentially wraps the [`Interface`] trait.
// fn as_interface<T: Interface>(&mut self) -> Option<T> {
// Interface::from_runtime(self)
// }
}

62
crates/plugin_runtime/src/wasi.rs
View file

@ -1,11 +1,10 @@
use std::collections::HashMap;
use std::{collections::HashMap, fs::File, path::Path};
use anyhow::anyhow;
use anyhow::{anyhow, Error};
use serde::{de::DeserializeOwned, Serialize};
use wasmtime::{Engine, Func, Instance, Linker, Memory, MemoryType, Module, Store, TypedFunc};
use wasmtime_wasi::{WasiCtx, WasiCtxBuilder};
use crate::*;
use wasmtime_wasi::{dir, Dir, WasiCtx, WasiCtxBuilder};
pub struct Wasi {
engine: Engine,
@ -17,7 +16,8 @@ pub struct Wasi {
}
pub struct WasiPlugin {
pub source_bytes: Vec<u8>,
pub module: Vec<u8>,
pub wasi_ctx: WasiCtx,
}
impl Wasi {
@ -39,24 +39,23 @@ impl Wasi {
}
}
impl Runtime for Wasi {
type Plugin = WasiPlugin;
type Error = anyhow::Error;
impl Wasi {
pub fn default_ctx() -> WasiCtx {
WasiCtxBuilder::new()
.inherit_stdout()
.inherit_stderr()
.build()
}
fn init(plugin: WasiPlugin) -> Result<Self, Self::Error> {
pub fn init(plugin: WasiPlugin) -> Result<Self, Error> {
let engine = Engine::default();
let mut linker = Linker::new(&engine);
println!("linking");
wasmtime_wasi::add_to_linker(&mut linker, |s| s)?;
println!("linked");
let wasi = WasiCtxBuilder::new()
.inherit_stdout()
.inherit_stderr()
.build();
let mut store: Store<_> = Store::new(&engine, wasi);
let mut store: Store<_> = Store::new(&engine, plugin.wasi_ctx);
println!("moduling");
let module = Module::new(&engine, plugin.source_bytes)?;
let module = Module::new(&engine, plugin.module)?;
println!("moduled");
linker.module(&mut store, "", &module)?;
@ -78,13 +77,20 @@ impl Runtime for Wasi {
})
}
// fn constant<T: DeserializeOwned>(&mut self, handle: &Handle) -> Result<T, Self::Error> {
// let export = self
// .instance
// .get_export(&mut self.store, handle.inner())
// .ok_or_else(|| anyhow!("Could not get export"))?;
pub fn attach_file<T: AsRef<Path>>(&mut self, path: T) -> Result<(), Error> {
let ctx = self.store.data_mut();
let file = File::open(&path).unwrap();
let dir = Dir::from_std_file(file);
// this is a footgun and a half.
let dir = dir::Dir::from_cap_std(dir);
ctx.push_preopened_dir(Box::new(dir), path)?;
Ok(())
}
// todo!()
// pub fn remove_file<T: AsRef<Path>>(&mut self, path: T) -> Result<(), Error> {
// let ctx = self.store.data_mut();
// ctx.remove
// Ok(())
// }
// So this call function is kinda a dance, I figured it'd be a good idea to document it.
@ -131,11 +137,11 @@ impl Runtime for Wasi {
// so the heap is still valid for our inspection when we want to pull things out.
// TODO: dont' use as for conversions
fn call<A: Serialize, R: DeserializeOwned>(
pub fn call<A: Serialize, R: DeserializeOwned>(
&mut self,
handle: &str,
arg: A,
) -> Result<R, Self::Error> {
) -> Result<R, Error> {
// serialize the argument using bincode
let arg = bincode::serialize(&arg)?;
let arg_buffer_len = arg.len();
@ -185,10 +191,4 @@ impl Runtime for Wasi {
return Ok(result);
}
// fn register_handle<T: AsRef<str>>(&mut self, name: T) -> bool {
// self.instance
// .get_export(&mut self.store, name.as_ref())
// .is_some()
// }
}

177
crates/plugin_runtime/src/wasm.rs
View file

@ -1,177 +0,0 @@
use std::collections::HashMap;
use anyhow::anyhow;
use wasmtime::{Engine, Func, Instance, Memory, MemoryType, Module, Store, TypedFunc};
use crate::*;
pub struct Wasm {
engine: Engine,
module: Module,
store: Store<()>,
instance: Instance,
alloc_buffer: TypedFunc<i32, i32>,
// free_buffer: TypedFunc<(i32, i32), ()>,
}
pub struct WasmPlugin {
pub source_bytes: Vec<u8>,
}
impl Wasm {
pub fn dump_memory(data: &[u8]) {
for (i, byte) in data.iter().enumerate() {
if i % 32 == 0 {
println!();
}
if i % 4 == 0 {
print!("|");
}
if *byte == 0 {
print!("__")
} else {
print!("{:02x}", byte);
}
}
println!();
}
}
impl Runtime for Wasm {
type Plugin = WasmPlugin;
type Error = anyhow::Error;
fn init(plugin: WasmPlugin) -> Result<Self, Self::Error> {
let engine = Engine::default();
let module = Module::new(&engine, plugin.source_bytes)?;
let mut store: Store<()> = Store::new(&engine, ());
let instance = Instance::new(&mut store, &module, &[])?;
let alloc_buffer = instance.get_typed_func(&mut store, "__alloc_buffer")?;
// let free_buffer = instance.get_typed_func(&mut store, "__free_buffer")?;
Ok(Wasm {
engine,
module,
store,
instance,
alloc_buffer,
// free_buffer,
})
}
// fn constant<T: DeserializeOwned>(&mut self, handle: &Handle) -> Result<T, Self::Error> {
// let export = self
// .instance
// .get_export(&mut self.store, handle.inner())
// .ok_or_else(|| anyhow!("Could not get export"))?;
// todo!()
// }
// So this call function is kinda a dance, I figured it'd be a good idea to document it.
// the high level is we take a serde type, serialize it to a byte array,
// (we're doing this using bincode for now)
// then toss that byte array into webassembly.
// webassembly grabs that byte array, does some magic,
// and serializes the result into yet another byte array.
// we then grab *that* result byte array and deserialize it into a result.
//
// phew...
//
// now the problem is, webassambly doesn't support buffers.
// only really like i32s, that's it (yeah, it's sad. Not even unsigned!)
// (ok, I'm exaggerating a bit).
//
// the Wasm function that this calls must have a very specific signature:
//
// fn(pointer to byte array: i32, length of byte array: i32)
// -> pointer to (
// pointer to byte_array: i32,
// length of byte array: i32,
// ): i32
//
// This pair `(pointer to byte array, length of byte array)` is called a `Buffer`
// and can be found in the cargo_test plugin.
//
// so on the wasm side, we grab the two parameters to the function,
// stuff them into a `Buffer`,
// and then pray to the `unsafe` Rust gods above that a valid byte array pops out.
//
// On the flip side, when returning from a wasm function,
// we convert whatever serialized result we get into byte array,
// which we stuff into a Buffer and allocate on the heap,
// which pointer to we then return.
// Note the double indirection!
//
// So when returning from a function, we actually leak memory *twice*:
//
// 1) once when we leak the byte array
// 2) again when we leak the allocated `Buffer`
//
// This isn't a problem because Wasm stops executing after the function returns,
// so the heap is still valid for our inspection when we want to pull things out.
// TODO: dont' use as for conversions
fn call<A: Serialize, R: DeserializeOwned>(
&mut self,
handle: &str,
arg: A,
) -> Result<R, Self::Error> {
// serialize the argument using bincode
let arg = bincode::serialize(&arg)?;
let arg_buffer_len = arg.len();
// allocate a buffer and write the argument to that buffer
let arg_buffer_ptr = self
.alloc_buffer
.call(&mut self.store, arg_buffer_len as i32)?;
let plugin_memory = self
.instance
.get_memory(&mut self.store, "memory")
.ok_or_else(|| anyhow!("Could not grab slice of plugin memory"))?;
plugin_memory.write(&mut self.store, arg_buffer_ptr as usize, &arg)?;
// get the webassembly function we want to actually call
// TODO: precompute handle
let fun_name = format!("__{}", handle);
let fun = self
.instance
.get_typed_func::<(i32, i32), i32, _>(&mut self.store, &fun_name)?;
// call the function, passing in the buffer and its length
// this should return a pointer to a (ptr, lentgh) pair
let arg_buffer = (arg_buffer_ptr, arg_buffer_len as i32);
let result_buffer = fun.call(&mut self.store, arg_buffer)?;
// create a buffer to read the (ptr, length) pair into
// this is a total of 4 + 4 = 8 bytes.
let buffer = &mut [0; 8];
plugin_memory.read(&mut self.store, result_buffer as usize, buffer)?;
// use these bytes (wasm stores things little-endian)
// to get a pointer to the buffer and its length
let b = buffer;
let result_buffer_ptr = u32::from_le_bytes([b[0], b[1], b[2], b[3]]) as usize;
let result_buffer_len = u32::from_le_bytes([b[4], b[5], b[6], b[7]]) as usize;
let result_buffer_end = result_buffer_ptr + result_buffer_len;
// read the buffer at this point into a byte array
// deserialize the byte array into the provided serde type
let result = &plugin_memory.data(&mut self.store)[result_buffer_ptr..result_buffer_end];
let result = bincode::deserialize(result)?;
// TODO: this is handled wasm-side, but I'd like to double-check
// // deallocate the argument buffer
// self.free_buffer.call(&mut self.store, arg_buffer);
return Ok(result);
}
// fn register_handle<T: AsRef<str>>(&mut self, name: T) -> bool {
// self.instance
// .get_export(&mut self.store, name.as_ref())
// .is_some()
// }
}