Macro system done in theory

too afraid to begin debugging, resting for a moment

.cargo/config.toml

@@ -9,3 +9,6 @@ ORCHID_EXTENSIONS = "target/debug/orchid-std"
 ORCHID_DEFAULT_SYSTEMS = "orchid::std"
 ORCHID_LOG_BUFFERS = "true"
 RUSTBACKTRACE = "1"
+
+[build]
+# rustflags = ["-Znext-solver"]

notes/expr_refs.md

@@ -6,13 +6,13 @@ Reference loops are resource leaks. There are two primary ways to avoid referenc
 - Constants reference their constituent Expressions
 - Expressions reference Atoms
 - During evaluation, Constants replace their unbound names with Constants
-  - There is a reference cycle here, but it always goes through a Constant.
+	- There is a reference cycle here, but it always goes through a Constant.
-    > **todo** A potential fix may be to update all Constants to point to a dummy value before freeing Trees
+		> **todo** A potential fix may be to update all Constants to point to a dummy value before freeing Trees
 - Atoms reference the Systems that implement them
 - Atoms may reference Expressions that are not younger than them
-  - This link is managed by the System but tied to Atom and not System lifecycle
+	- This link is managed by the System but tied to Atom and not System lifecycle
-  - Atoms can technically be applied to themselves, but it's a copying apply so it probably isn't a risk factor
+	- Atoms can technically be applied to themselves, but it's a copying apply so it probably isn't a risk factor
 - Systems reference the Extension that contains them
 - Extensions reference the Port that connects them
-  - The Extension signals the remote peer to disconnect on drop
+	- The Extension signals the remote peer to disconnect on drop
-  - The port is also referenced in a loose receiver thread, which always eventually tries to find the Extension or polls for ingress so it always eventually exits after the Extension's drop handler is called
+	- The port is also referenced in a loose receiver thread, which always eventually tries to find the Extension or polls for ingress so it always eventually exits after the Extension's drop handler is called

notes/macros.md

@@ -17,11 +17,11 @@ Priority numbers are written in hexadecimal normal form to avoid precision bugs,
 - **32-39**: Binary operators, in inverse priority order
 - **80-87**: Expression-like structures such as if/then/else
 - **128-135**: Anything that creates lambdas
-  Programs triggered by a lower priority pattern than this can assume that all names are correctly bound
+	Programs triggered by a lower priority pattern than this can assume that all names are correctly bound
 - **200**: Aliases extracted for readability
-  The user-accessible entry points of all macro programs must be lower priority than this, so any arbitrary syntax can be extracted into an alias with no side effects
+	The user-accessible entry points of all macro programs must be lower priority than this, so any arbitrary syntax can be extracted into an alias with no side effects
 - **224-231**: Integration; documented hooks exposed by a macro package to allow third party packages to extend its functionality
-  The `statement` pattern produced by `do{}` blocks and matched by `let` and `cps` is a good example of this. When any of these are triggered, all macro programs are in a documented state.
+	The `statement` pattern produced by `do{}` blocks and matched by `let` and `cps` is a good example of this. When any of these are triggered, all macro programs are in a documented state.
 - **248-255**: Transitional states within macro programs get the highest priority
 
 The numbers are arbitrary and up for debate. These are just the ones I came up with when writing the examples.

notes/new_macro_model.md

@@ -42,9 +42,9 @@ Prioritised macro patterns must start and end with a vectorial placeholder. They
 Macros are checked from the outermost block inwards.
 
 1. For every name token, test all named macros starting with that name
-   1. If the tail is implicit, continue iterating
+	 1. If the tail is implicit, continue iterating
 2. Test all prioritized macros
-   1. Take the first rule that matches in the highest prioritized block
+	 1. Take the first rule that matches in the highest prioritized block
 
 Test all in a set of macros
 1. Take the first rule that matches in each block
@@ -75,26 +75,26 @@ Recursion has to happen through the interpreter itself, so the macro system is d
 - line parser `macro` parses a macro with the existing logic
 - atom `MacRecurState` holds the recursion state
 - function `resolve_recur` finds all matches on a MacTree
-  - type: `MacRecurState -> MacTree -> MacTree`
+	- type: `MacRecurState -> MacTree -> MacTree`
-  - use all relevant macros to find all matches in the tree
+	- use all relevant macros to find all matches in the tree
-    - since macros must contain a locally defined token, it can be assumed that at the point that a constant is evaluated and all imports in the parent module have been resolved, necessarily all relevant macro rules must have been loaded
+		- since macros must contain a locally defined token, it can be assumed that at the point that a constant is evaluated and all imports in the parent module have been resolved, necessarily all relevant macro rules must have been loaded
-  - for each match
+	- for each match
-    - check for recursion violations
+		- check for recursion violations
-    - wrap the body in iife-s corresponding to the named values in the match state
+		- wrap the body in iife-s corresponding to the named values in the match state
-    - emit a recursive call to process and run the body, and pass the same recursive call as argument for the macro to use
+		- emit a recursive call to process and run the body, and pass the same recursive call as argument for the macro to use
 		```
 		(\recur. lower (recur $body) recur)
 			(resolve_recur $mac_recur_state)
 		```
-  - emit a single call to `instantiate_tpl` which receives all of these
+	- emit a single call to `instantiate_tpl` which receives all of these
 - function `instantiate_tpl` inserts `MacTree` values into a `MacTree(tpl)`
-  - type: `MacTree(tpl) [-> MacTree] -> MacTree`  
+	- type: `MacTree(tpl) [-> MacTree] -> MacTree`  
 		_this function deduces the number of arguments from the first argument. This combines poorly with autocurry, but it's an easy way to avoid representing standalone tree lists_
-  - walks the tree to find max template slot number, reads and type checks as many template values
+	- walks the tree to find max template slot number, reads and type checks as many template values
-  - returns the populated tree
+	- returns the populated tree
 - function `resolve` is the main entry point of the code
-  - type: `MacTree -> MacTree`
+	- type: `MacTree -> MacTree`
-  - invokes `resolve_recur` with an empty `MacRecurState`
+	- invokes `resolve_recur` with an empty `MacRecurState`
 - function `lower` is the main exit point of the code
-  - type: `MacTree -> any`
+	- type: `MacTree -> any`
-  - Lowers `MacTree` into the equivalent `Expr`.
+	- Lowers `MacTree` into the equivalent `Expr`.

orchid-api/src/parser.rs

@@ -85,8 +85,8 @@ pub struct Comment {
 /// called during a [FetchParsedConst] call, but it can be called for a
 /// different [ParsedConstId] from the one in [FetchParsedConst].
 ///
-/// Each name is either resolved to an alias or existing constant `Some(TStrv)`
+/// Each name is either resolved to a valid name or a potential error error.
-/// or not resolved `None`. An error is never raised, as names may have a
+/// The error is not raised by the interpreter itself, as names may have a
 /// primary meaning such as a local binding which can be overridden by specific
 /// true names such as those triggering macro keywords. It is not recommended to
 /// define syntax that can break by defining arbitrary constants, as line
@@ -100,5 +100,5 @@ pub struct ResolveNames {
 }
 
 impl Request for ResolveNames {
-	type Response = Vec<Option<TStrv>>;
+	type Response = Vec<OrcResult<TStrv>>;
 }

orchid-base/src/clone.rs

@@ -6,7 +6,7 @@ macro_rules! clone {
       $body
     }
   );
-  ($($n:ident),+) => {
+  ($($n:ident $($mut:ident)?),+) => {
-    $( let $n = $n.clone(); )+
+    $( let $($mut)? $n = $n.clone(); )+
   }
 }

orchid-base/src/error.rs

@@ -71,9 +71,8 @@ impl OrcErr {
 		}
 	}
 }
-impl Eq for OrcErr {}
+impl PartialEq<Tok<String>> for OrcErr {
-impl PartialEq for OrcErr {
+	fn eq(&self, other: &Tok<String>) -> bool { self.description == *other }
-	fn eq(&self, other: &Self) -> bool { self.description == other.description }
 }
 impl From<OrcErr> for Vec<OrcErr> {
 	fn from(value: OrcErr) -> Self { vec![value] }
@@ -192,16 +191,8 @@ macro_rules! join_ok {
 	(@VALUES) => { Ok(()) };
 }
 
-pub fn mk_err(
+pub fn mk_errv_floating(description: Tok<String>, message: impl AsRef<str>) -> OrcErrv {
-	description: Tok<String>,
+	mk_errv::<Pos>(description, message, [])
-	message: impl AsRef<str>,
-	posv: impl IntoIterator<Item = ErrPos>,
-) -> OrcErr {
-	OrcErr {
-		description,
-		message: Arc::new(message.as_ref().to_string()),
-		positions: posv.into_iter().collect(),
-	}
 }
 
 pub fn mk_errv<I: Into<ErrPos>>(
@@ -209,7 +200,12 @@ pub fn mk_errv<I: Into<ErrPos>>(
 	message: impl AsRef<str>,
 	posv: impl IntoIterator<Item = I>,
 ) -> OrcErrv {
-	mk_err(description, message, posv.into_iter().map_into()).into()
+	OrcErr {
+		description,
+		message: Arc::new(message.as_ref().to_string()),
+		positions: posv.into_iter().map_into().collect(),
+	}
+	.into()
 }
 
 pub async fn async_io_err<I: Into<ErrPos>>(

orchid-base/src/iter_utils.rs

@@ -0,0 +1,24 @@
+use std::fmt;
+
+use itertools::{Itertools, Position};
+
+pub struct PrintList<'a, I: Iterator<Item = E> + Clone, E: fmt::Display>(pub I, pub &'a str);
+impl<'a, I: Iterator<Item = E> + Clone, E: fmt::Display> fmt::Display for PrintList<'a, I, E> {
+	fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+		for (pos, item) in self.0.clone().with_position() {
+			match pos {
+				Position::First | Position::Only => write!(f, "{item}")?,
+				Position::Middle => write!(f, ", {item}")?,
+				Position::Last => write!(f, ", {} {item}", self.1)?,
+			}
+		}
+		Ok(())
+	}
+}
+
+pub trait IteratorPrint: Iterator<Item: fmt::Display> + Clone {
+	fn display<'a>(self, operator: &'a str) -> PrintList<'a, Self, Self::Item> {
+		PrintList(self, operator)
+	}
+}
+impl<T: Iterator<Item: fmt::Display> + Clone> IteratorPrint for T {}

orchid-base/src/lib.rs

@@ -12,6 +12,7 @@ pub mod event;
 pub mod format;
 pub mod id_store;
 pub mod interner;
+pub mod iter_utils;
 pub mod join;
 pub mod location;
 pub mod logging;

orchid-base/src/match_mapping.rs

@@ -1,4 +1,4 @@
-/// A shorthand for mapping over enums with identical structure. Used for
+/// A shorthand for mapping over enums with similar structure. Used for
 /// converting between owned enums and the corresponding API enums that only
 /// differ in the type of their fields.
 ///
@@ -7,7 +7,11 @@
 /// match_mapping!(self, ThisType => OtherType {
 ///   EmptyVariant,
 ///   TupleVariant(foo => intern(foo), bar.clone()),
-///   StructVariant{ a.to_api(), b }
+///   StructVariant{ a.to_api(), b },
+///   DedicatedConverter(value () convert)
+/// } {
+///   ThisType::DimorphicVariant(c) => OtherType::CorrespondingVariant(c.left(), c.right()),
+///   ThisType::UnexpectedVariant => panic!(),
 /// })
 /// ```
 #[macro_export]

orchid-base/src/name.rs

@@ -19,10 +19,9 @@ trait_set! {
 	pub trait NameIter = Iterator<Item = Tok<String>> + DoubleEndedIterator + ExactSizeIterator;
 }
 
-/// A token path which may be empty. [VName] is the non-empty,
+/// A token path which may be empty. [VName] is the non-empty version
-/// [PathSlice] is the borrowed version
 #[derive(Clone, Default, Hash, PartialEq, Eq)]
-pub struct VPath(pub Vec<Tok<String>>);
+pub struct VPath(Vec<Tok<String>>);
 impl VPath {
 	/// Collect segments into a vector
 	pub fn new(items: impl IntoIterator<Item = Tok<String>>) -> Self {

orchid-base/src/number.rs

@@ -3,7 +3,7 @@ use std::ops::Range;
 
 use ordered_float::NotNan;
 
-use crate::error::{OrcErr, mk_err};
+use crate::error::{OrcErrv, mk_errv};
 use crate::interner::Interner;
 use crate::location::SrcRange;
 use crate::name::Sym;
@@ -55,20 +55,20 @@ pub struct NumError {
 	pub kind: NumErrorKind,
 }
 
-pub async fn num_to_err(
+pub async fn num_to_errv(
 	NumError { kind, range }: NumError,
 	offset: u32,
 	source: &Sym,
 	i: &Interner,
-) -> OrcErr {
+) -> OrcErrv {
-	mk_err(
+	mk_errv(
 		i.i("Failed to parse number").await,
 		match kind {
 			NumErrorKind::NaN => "NaN emerged during parsing",
 			NumErrorKind::InvalidDigit => "non-digit character encountered",
 			NumErrorKind::Overflow => "The number being described is too large or too accurate",
 		},
-		[SrcRange::new(offset + range.start as u32..offset + range.end as u32, source).pos().into()],
+		[SrcRange::new(offset + range.start as u32..offset + range.end as u32, source)],
 	)
 }
 
@@ -92,11 +92,11 @@ pub fn parse_num(string: &str) -> Result<Numeric, NumError> {
 	match base_s.split_once('.') {
 		None => {
 			let base = int_parse(base_s, radix, pos)?;
-			if let Ok(pos_exp) = u32::try_from(exponent) {
+			if let Ok(pos_exp) = u32::try_from(exponent)
-				if let Some(radical) = u64::from(radix).checked_pow(pos_exp) {
+				&& let Some(radical) = u64::from(radix).checked_pow(pos_exp)
-					let num = base.checked_mul(radical).and_then(|m| m.try_into().ok()).ok_or(overflow_e)?;
+			{
-					return Ok(Numeric::Int(num));
+				let num = base.checked_mul(radical).and_then(|m| m.try_into().ok()).ok_or(overflow_e)?;
-				}
+				return Ok(Numeric::Int(num));
 			}
 			let f = (base as f64) * (radix as f64).powi(exponent);
 			let err = NumError { range: 0..string.len(), kind: NumErrorKind::NaN };

orchid-base/src/parse.rs

@@ -7,11 +7,11 @@ use futures::future::join_all;
 use itertools::Itertools;
 
 use crate::api;
-use crate::error::{OrcErrv, OrcRes, Reporter, mk_err, mk_errv};
+use crate::error::{OrcErrv, OrcRes, Reporter, mk_errv};
 use crate::format::{FmtCtx, FmtUnit, Format, fmt};
 use crate::interner::{Interner, Tok};
 use crate::location::SrcRange;
-use crate::name::{NameLike, Sym, VName, VPath};
+use crate::name::{Sym, VName, VPath};
 use crate::tree::{ExprRepr, ExtraTok, Paren, TokTree, Token, ttv_fmt, ttv_range};
 
 pub trait ParseCtx {
@@ -237,10 +237,10 @@ pub async fn parse_multiname<'a, A: ExprRepr, X: ExtraTok>(
 		match &tt.tok {
 			Token::NS(ns, body) => {
 				if !ns.starts_with(name_start) {
-					ctx.rep().report(mk_err(
+					ctx.rep().report(mk_errv(
 						ctx.i().i("Unexpected name prefix").await,
 						"Only names can precede ::",
-						[ttpos.into()],
+						[ttpos],
 					))
 				};
 				let out = Box::pin(rec(body, ctx)).await?;

orchid-extension/src/atom.rs

@@ -15,7 +15,7 @@ use futures::{FutureExt, StreamExt};
 use orchid_api_derive::Coding;
 use orchid_api_traits::{Coding, Decode, Encode, Request, enc_vec};
 use orchid_base::clone;
-use orchid_base::error::{OrcErr, OrcRes, mk_err};
+use orchid_base::error::{OrcErrv, OrcRes, mk_errv, mk_errv_floating};
 use orchid_base::format::{FmtCtx, FmtUnit, Format};
 use orchid_base::interner::Interner;
 use orchid_base::location::Pos;
@@ -24,6 +24,7 @@ use orchid_base::reqnot::Requester;
 use trait_set::trait_set;
 
 use crate::api;
+use crate::conv::ToExpr;
 // use crate::error::{ProjectError, ProjectResult};
 use crate::expr::{Expr, ExprData, ExprHandle, ExprKind};
 use crate::gen_expr::GExpr;
@@ -92,7 +93,7 @@ pub struct ForeignAtom {
 }
 impl ForeignAtom {
 	pub fn pos(&self) -> Pos { self.pos.clone() }
-	pub fn ctx(&self) -> SysCtx { self.expr.ctx.clone() }
+	pub fn ctx(&self) -> &SysCtx { &self.expr.ctx }
 	pub fn ex(self) -> Expr {
 		let (handle, pos) = (self.expr.clone(), self.pos.clone());
 		let data = ExprData { pos, kind: ExprKind::Atom(ForeignAtom { ..self }) };
@@ -110,6 +111,9 @@ impl ForeignAtom {
 		.await?;
 		Some(M::Response::decode(Pin::new(&mut &rep[..])).await)
 	}
+	pub async fn downcast<T: AtomicFeatures>(self) -> Result<TypAtom<T>, NotTypAtom> {
+		TypAtom::downcast(self.ex().handle()).await
+	}
 }
 impl fmt::Display for ForeignAtom {
 	fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { write!(f, "Atom::{:?}", self.atom) }
@@ -122,6 +126,9 @@ impl Format for ForeignAtom {
 		FmtUnit::from_api(&self.ctx().reqnot().request(api::ExtAtomPrint(self.atom.clone())).await)
 	}
 }
+impl ToExpr for ForeignAtom {
+	async fn to_expr(self) -> GExpr { self.ex().to_expr().await }
+}
 
 pub struct NotTypAtom {
 	pub pos: Pos,
@@ -130,11 +137,11 @@ pub struct NotTypAtom {
 	pub ctx: SysCtx,
 }
 impl NotTypAtom {
-	pub async fn mk_err(&self) -> OrcErr {
+	pub async fn mk_err(&self) -> OrcErrv {
-		mk_err(
+		mk_errv(
 			self.ctx.i().i("Not the expected type").await,
 			format!("This expression is not a {}", self.typ.name()),
-			[self.pos.clone().into()],
+			[self.pos.clone()],
 		)
 	}
 }
@@ -216,10 +223,12 @@ impl<A: AtomCard> Default for MethodSetBuilder<A> {
 
 #[derive(Clone)]
 pub struct TypAtom<A: AtomicFeatures> {
-	pub data: ForeignAtom,
+	pub untyped: ForeignAtom,
 	pub value: A::Data,
 }
 impl<A: AtomicFeatures> TypAtom<A> {
+	pub fn ctx(&self) -> &SysCtx { self.untyped.ctx() }
+	pub fn i(&self) -> &Interner { self.ctx().i() }
 	pub async fn downcast(expr: Rc<ExprHandle>) -> Result<Self, NotTypAtom> {
 		match Expr::from_handle(expr).atom().await {
 			Err(expr) => Err(NotTypAtom {
@@ -242,9 +251,9 @@ impl<A: AtomicFeatures> TypAtom<A> {
 	pub async fn request<M: AtomMethod>(&self, req: M) -> M::Response
 	where A: Supports<M> {
 		M::Response::decode(Pin::new(
-			&mut &(self.data.ctx().reqnot().request(api::Fwd(
+			&mut &(self.untyped.ctx().reqnot().request(api::Fwd(
-				self.data.atom.clone(),
+				self.untyped.atom.clone(),
-				Sym::parse(M::NAME, self.data.ctx().i()).await.unwrap().tok().to_api(),
+				Sym::parse(M::NAME, self.untyped.ctx().i()).await.unwrap().tok().to_api(),
 				enc_vec(&req).await,
 			)))
 			.await
@@ -257,6 +266,9 @@ impl<A: AtomicFeatures> Deref for TypAtom<A> {
 	type Target = A::Data;
 	fn deref(&self) -> &Self::Target { &self.value }
 }
+impl<A: AtomicFeatures> ToExpr for TypAtom<A> {
+	async fn to_expr(self) -> GExpr { self.untyped.to_expr().await }
+}
 
 pub struct AtomCtx<'a>(pub &'a [u8], pub Option<api::AtomId>, pub SysCtx);
 impl FmtCtx for AtomCtx<'_> {
@@ -317,10 +329,10 @@ impl Format for AtomFactory {
 	}
 }
 
-pub async fn err_not_callable(i: &Interner) -> OrcErr {
+pub async fn err_not_callable(i: &Interner) -> OrcErrv {
-	mk_err(i.i("This atom is not callable").await, "Attempted to apply value as function", [])
+	mk_errv_floating(i.i("This atom is not callable").await, "Attempted to apply value as function")
 }
 
-pub async fn err_not_command(i: &Interner) -> OrcErr {
+pub async fn err_not_command(i: &Interner) -> OrcErrv {
-	mk_err(i.i("This atom is not a command").await, "Settled on an inactionable value", [])
+	mk_errv_floating(i.i("This atom is not a command").await, "Settled on an inactionable value")
 }

orchid-extension/src/atom_owned.rs

@@ -217,7 +217,7 @@ pub trait OwnedAtom: Atomic<Variant = OwnedVariant> + Any + Clone + 'static {
 	fn val(&self) -> impl Future<Output = Cow<'_, Self::Data>>;
 	#[allow(unused_variables)]
 	fn call_ref(&self, arg: Expr) -> impl Future<Output = GExpr> {
-		async move { bot([err_not_callable(arg.ctx().i()).await]) }
+		async move { bot(err_not_callable(arg.ctx().i()).await) }
 	}
 	fn call(self, arg: Expr) -> impl Future<Output = GExpr> {
 		async {
@@ -229,12 +229,12 @@ pub trait OwnedAtom: Atomic<Variant = OwnedVariant> + Any + Clone + 'static {
 	}
 	#[allow(unused_variables)]
 	fn command(self, ctx: SysCtx) -> impl Future<Output = OrcRes<Option<GExpr>>> {
-		async move { Err(err_not_command(ctx.i()).await.into()) }
+		async move { Err(err_not_command(ctx.i()).await) }
 	}
 	#[allow(unused_variables)]
 	fn free(self, ctx: SysCtx) -> impl Future<Output = ()> { async {} }
 	#[allow(unused_variables)]
-	fn print<'a>(&'a self, c: &'a (impl FmtCtx + ?Sized + 'a)) -> impl Future<Output = FmtUnit> {
+	fn print_atom<'a>(&'a self, c: &'a (impl FmtCtx + ?Sized + 'a)) -> impl Future<Output = FmtUnit> {
 		async { format!("OwnedAtom({})", type_name::<Self>()).into() }
 	}
 	#[allow(unused_variables)]
@@ -294,7 +294,7 @@ impl<T: OwnedAtom> DynOwnedAtom for T {
 		self.free(ctx).boxed_local()
 	}
 	fn dyn_print(&self, ctx: SysCtx) -> LocalBoxFuture<'_, FmtUnit> {
-		async move { self.print(&FmtCtxImpl { i: ctx.i() }).await }.boxed_local()
+		async move { self.print_atom(&FmtCtxImpl { i: ctx.i() }).await }.boxed_local()
 	}
 	fn dyn_serialize<'a>(
 		&'a self,
@@ -315,10 +315,10 @@ struct ObjStore {
 }
 impl SysCtxEntry for ObjStore {}
 
-pub async fn get_own_instance<A: OwnedAtom>(typ: TypAtom<A>) -> A {
+pub async fn own<A: OwnedAtom>(typ: TypAtom<A>) -> A {
-	let ctx = typ.data.ctx();
+	let ctx = typ.untyped.ctx();
 	let g = ctx.get_or_default::<ObjStore>().objects.read().await;
-	let dyn_atom = (g.get(&typ.data.atom.drop.expect("Owned atoms always have a drop ID")))
+	let dyn_atom = (g.get(&typ.untyped.atom.drop.expect("Owned atoms always have a drop ID")))
 		.expect("Atom ID invalid; atom type probably not owned by this crate");
 	dyn_atom.as_any_ref().downcast_ref().cloned().expect("The ID should imply a type as well")
 }

orchid-extension/src/atom_thin.rs

@@ -105,11 +105,11 @@ pub trait ThinAtom:
 {
 	#[allow(unused_variables)]
 	fn call(&self, arg: Expr) -> impl Future<Output = GExpr> {
-		async move { bot([err_not_callable(arg.ctx().i()).await]) }
+		async move { bot(err_not_callable(arg.ctx().i()).await) }
 	}
 	#[allow(unused_variables)]
 	fn command(&self, ctx: SysCtx) -> impl Future<Output = OrcRes<Option<GExpr>>> {
-		async move { Err(err_not_command(ctx.i()).await.into()) }
+		async move { Err(err_not_command(ctx.i()).await) }
 	}
 	#[allow(unused_variables)]
 	fn print(&self, ctx: SysCtx) -> impl Future<Output = FmtUnit> {

orchid-extension/src/conv.rs

@@ -1,11 +1,11 @@
 use std::future::Future;
 
 use never::Never;
-use orchid_base::error::{OrcErr, OrcRes, mk_err};
+use orchid_base::error::{OrcErrv, OrcRes, mk_errv};
 use orchid_base::interner::Interner;
 use orchid_base::location::Pos;
 
-use crate::atom::{AtomicFeatures, ToAtom, TypAtom};
+use crate::atom::{AtomicFeatures, ForeignAtom, ToAtom, TypAtom};
 use crate::expr::Expr;
 use crate::gen_expr::{GExpr, atom, bot};
 use crate::system::{SysCtx, downcast_atom};
@@ -24,22 +24,29 @@ impl<T: TryFromExpr, U: TryFromExpr> TryFromExpr for (T, U) {
 	}
 }
 
-async fn err_not_atom(pos: Pos, i: &Interner) -> OrcErr {
+async fn err_not_atom(pos: Pos, i: &Interner) -> OrcErrv {
-	mk_err(i.i("Expected an atom").await, "This expression is not an atom", [pos.into()])
+	mk_errv(i.i("Expected an atom").await, "This expression is not an atom", [pos])
 }
 
-async fn err_type(pos: Pos, i: &Interner) -> OrcErr {
+async fn err_type(pos: Pos, i: &Interner) -> OrcErrv {
-	mk_err(i.i("Type error").await, "The atom is a different type than expected", [pos.into()])
+	mk_errv(i.i("Type error").await, "The atom is a different type than expected", [pos])
+}
+
+impl TryFromExpr for ForeignAtom {
+	async fn try_from_expr(expr: Expr) -> OrcRes<Self> {
+		match expr.atom().await {
+			Err(ex) => Err(err_not_atom(ex.data().await.pos.clone(), ex.ctx().i()).await),
+			Ok(f) => Ok(f),
+		}
+	}
 }
 
 impl<A: AtomicFeatures> TryFromExpr for TypAtom<A> {
 	async fn try_from_expr(expr: Expr) -> OrcRes<Self> {
-		match expr.atom().await {
+		let f = ForeignAtom::try_from_expr(expr).await?;
-			Err(ex) => Err(err_not_atom(ex.data().await.pos.clone(), ex.ctx().i()).await.into()),
+		match downcast_atom::<A>(f).await {
-			Ok(f) => match downcast_atom::<A>(f).await {
+			Ok(a) => Ok(a),
-				Ok(a) => Ok(a),
+			Err(f) => Err(err_type(f.pos(), f.ctx().i()).await),
-				Err(f) => Err(err_type(f.pos(), f.ctx().i()).await.into()),
-			},
 		}
 	}
 }
@@ -49,29 +56,29 @@ impl TryFromExpr for SysCtx {
 }
 
 pub trait ToExpr {
-	fn to_expr(self) -> GExpr;
+	fn to_expr(self) -> impl Future<Output = GExpr>;
 }
 
 impl ToExpr for GExpr {
-	fn to_expr(self) -> GExpr { self }
+	async fn to_expr(self) -> GExpr { self }
 }
 impl ToExpr for Expr {
-	fn to_expr(self) -> GExpr { self.slot() }
+	async fn to_expr(self) -> GExpr { self.slot() }
 }
 
 impl<T: ToExpr> ToExpr for OrcRes<T> {
-	fn to_expr(self) -> GExpr {
+	async fn to_expr(self) -> GExpr {
 		match self {
 			Err(e) => bot(e),
-			Ok(t) => t.to_expr(),
+			Ok(t) => t.to_expr().await,
 		}
 	}
 }
 
 impl<A: ToAtom> ToExpr for A {
-	fn to_expr(self) -> GExpr { atom(self) }
+	async fn to_expr(self) -> GExpr { atom(self) }
 }
 
 impl ToExpr for Never {
-	fn to_expr(self) -> GExpr { match self {} }
+	async fn to_expr(self) -> GExpr { match self {} }
 }

orchid-extension/src/coroutine_exec.rs

@@ -0,0 +1,96 @@
+use std::borrow::Cow;
+use std::marker::PhantomData;
+use std::pin::Pin;
+use std::rc::Rc;
+
+use async_std::channel::{Receiver, RecvError, Sender, bounded};
+use async_std::sync::Mutex;
+use futures::future::Fuse;
+use futures::{FutureExt, select};
+use never::Never;
+use orchid_base::error::OrcRes;
+
+use crate::atom::Atomic;
+use crate::atom_owned::{OwnedAtom, OwnedVariant};
+use crate::conv::{ToExpr, TryFromExpr};
+use crate::expr::Expr;
+use crate::gen_expr::{GExpr, arg, call, lambda, seq};
+
+enum Command {
+	Execute(GExpr, Sender<Expr>),
+	Register(GExpr, Sender<Expr>),
+}
+
+struct BuilderCoroutineData {
+	work: Mutex<Fuse<Pin<Box<dyn Future<Output = GExpr>>>>>,
+	cmd_recv: Receiver<Command>,
+}
+#[derive(Clone)]
+struct BuilderCoroutine(Rc<BuilderCoroutineData>);
+impl BuilderCoroutine {
+	pub async fn run(self) -> GExpr {
+		let cmd = {
+			let mut work = self.0.work.lock().await;
+			select! {
+				ret_val = &mut *work => { return ret_val },
+				cmd_res = self.0.cmd_recv.recv().fuse() => match cmd_res {
+					Ok(cmd) => cmd,
+					Err(RecvError) => return (&mut *work).await
+				},
+			}
+		};
+		match cmd {
+			Command::Execute(expr, reply) => call([
+				lambda(0, [seq([
+					arg(0),
+					call([Replier { reply, builder: self }.to_expr().await, arg(0)]),
+				])]),
+				expr,
+			]),
+			Command::Register(expr, reply) =>
+				call([Replier { reply, builder: self }.to_expr().await, expr]),
+		}
+	}
+}
+
+#[derive(Clone)]
+pub struct Replier {
+	reply: Sender<Expr>,
+	builder: BuilderCoroutine,
+}
+impl Atomic for Replier {
+	type Data = ();
+	type Variant = OwnedVariant;
+}
+impl OwnedAtom for Replier {
+	type Refs = Never;
+	async fn val(&self) -> Cow<'_, Self::Data> { Cow::Owned(()) }
+	async fn call(self, arg: Expr) -> GExpr {
+		let _ = self.reply.send(arg).await;
+		self.builder.run().await
+	}
+}
+
+pub async fn exec<R: ToExpr>(f: impl for<'a> AsyncFnOnce(ExecHandle<'a>) -> R + 'static) -> GExpr {
+	let (cmd_snd, cmd_recv) = bounded(1);
+	let work =
+		async { f(ExecHandle(cmd_snd, PhantomData)).await.to_expr().await }.boxed_local().fuse();
+	let coro = BuilderCoroutine(Rc::new(BuilderCoroutineData { cmd_recv, work: Mutex::new(work) }));
+	coro.run().await
+}
+
+static WEIRD_DROP_ERR: &str = "Coroutine dropped while we are being polled somehow";
+
+pub struct ExecHandle<'a>(Sender<Command>, PhantomData<&'a ()>);
+impl ExecHandle<'_> {
+	pub async fn exec<T: TryFromExpr>(&mut self, val: impl ToExpr) -> OrcRes<T> {
+		let (reply_snd, reply_recv) = bounded(1);
+		self.0.send(Command::Execute(val.to_expr().await, reply_snd)).await.expect(WEIRD_DROP_ERR);
+		T::try_from_expr(reply_recv.recv().await.expect(WEIRD_DROP_ERR)).await
+	}
+	pub async fn register(&mut self, val: impl ToExpr) -> Expr {
+		let (reply_snd, reply_recv) = bounded(1);
+		self.0.send(Command::Register(val.to_expr().await, reply_snd)).await.expect(WEIRD_DROP_ERR);
+		reply_recv.recv().await.expect(WEIRD_DROP_ERR)
+	}
+}

orchid-extension/src/entrypoint.rs

@@ -31,7 +31,7 @@ use crate::api;
 use crate::atom::{AtomCtx, AtomDynfo, AtomTypeId};
 use crate::atom_owned::take_atom;
 use crate::expr::{Expr, ExprHandle};
-use crate::lexer::{LexContext, err_cascade, err_not_applicable};
+use crate::lexer::{LexContext, ekey_cascade, ekey_not_applicable};
 use crate::parser::{PTok, PTokTree, ParsCtx, get_const, linev_into_api};
 use crate::system::{SysCtx, atom_by_idx};
 use crate::system_ctor::{CtedObj, DynSystemCtor};
@@ -230,16 +230,17 @@ pub fn extension_init(
 							let sys_ctx = get_ctx(sys).await;
 							let text = Tok::from_api(text, &i).await;
 							let src = Sym::from_api(src, sys_ctx.i()).await;
-							let ctx = LexContext { id, pos, text: &text, src, ctx: sys_ctx.clone() };
+							let rep = Reporter::new();
+							let ctx = LexContext { id, pos, text: &text, src, ctx: sys_ctx.clone(), rep: &rep };
 							let trigger_char = text.chars().nth(pos as usize).unwrap();
-							let err_na = err_not_applicable(&i).await;
+							let ekey_na = ekey_not_applicable(&i).await;
-							let err_cascade = err_cascade(&i).await;
+							let ekey_cascade = ekey_cascade(&i).await;
 							let lexers = sys_ctx.cted().inst().dyn_lexers();
 							for lx in lexers.iter().filter(|l| char_filter_match(l.char_filter(), trigger_char)) {
 								match lx.lex(&text[pos as usize..], &ctx).await {
-									Err(e) if e.any(|e| *e == err_na) => continue,
+									Err(e) if e.any(|e| *e == ekey_na) => continue,
 									Err(e) => {
-										let eopt = e.keep_only(|e| *e != err_cascade).map(|e| Err(e.to_api()));
+										let eopt = e.keep_only(|e| *e != ekey_cascade).map(|e| Err(e.to_api()));
 										return hand.handle(&lex, &eopt).await;
 									},
 									Ok((s, expr)) => {

orchid-extension/src/func_atom.rs

@@ -1,3 +1,9 @@
+//! #TODO: redefine this in terms of [crate::coroutine_exec::exec]. Try
+//! differentiating between eager and lazy arguments. [ExprFunc] can remain with
+//! tweaks but other techniques probably must go.
+//!
+//! Notice also that Func must still be resumable
+use std::any::TypeId;
 use std::borrow::Cow;
 use std::collections::HashMap;
 use std::future::Future;
@@ -20,6 +26,7 @@ use trait_set::trait_set;
 use crate::atom::Atomic;
 use crate::atom_owned::{DeserializeCtx, OwnedAtom, OwnedVariant};
 use crate::conv::ToExpr;
+use crate::coroutine_exec::{ExecHandle, exec};
 use crate::expr::Expr;
 use crate::gen_expr::GExpr;
 use crate::system::{SysCtx, SysCtxEntry};
@@ -29,13 +36,37 @@ trait_set! {
 }
 
 pub trait ExprFunc<I, O>: Clone + 'static {
-	const ARITY: u8;
+	fn argtyps() -> &'static [TypeId];
-	fn apply(&self, v: Vec<Expr>) -> impl Future<Output = OrcRes<GExpr>>;
+	fn apply<'a>(&self, hand: ExecHandle<'a>, v: Vec<Expr>) -> impl Future<Output = OrcRes<GExpr>>;
 }
 
 #[derive(Default)]
-struct FunsCtx(Mutex<HashMap<Sym, (u8, Rc<dyn FunCB>)>>);
+struct FunsCtx(Mutex<HashMap<Sym, FunRecord>>);
 impl SysCtxEntry for FunsCtx {}
+#[derive(Clone)]
+struct FunRecord {
+	argtyps: &'static [TypeId],
+	fun: Rc<dyn FunCB>,
+}
+
+async fn process_args<I, O, F: ExprFunc<I, O>>(f: F) -> FunRecord {
+	let argtyps = F::argtyps();
+	let fun = Rc::new(move |v: Vec<Expr>| {
+		clone!(f, v mut);
+		exec(async move |mut hand| {
+			let mut norm_args = Vec::with_capacity(v.len());
+			for (expr, typ) in v.into_iter().zip(argtyps) {
+				if *typ != TypeId::of::<Expr>() {
+					norm_args.push(hand.exec(expr).await?);
+				}
+			}
+			f.apply(hand, norm_args).await
+		})
+		.map(Ok)
+		.boxed_local()
+	});
+	FunRecord { argtyps, fun }
+}
 
 /// An Atom representing a partially applied named native function. These
 /// partial calls are serialized into the name of the native function and the
@@ -46,23 +77,22 @@ impl SysCtxEntry for FunsCtx {}
 pub(crate) struct Fun {
 	path: Sym,
 	args: Vec<Expr>,
-	arity: u8,
+	record: FunRecord,
-	fun: Rc<dyn FunCB>,
 }
 impl Fun {
 	pub async fn new<I, O, F: ExprFunc<I, O>>(path: Sym, ctx: SysCtx, f: F) -> Self {
 		let funs: &FunsCtx = ctx.get_or_default();
 		let mut fung = funs.0.lock().await;
-		let fun = if let Some(x) = fung.get(&path) {
+		let record = if let Some(record) = fung.get(&path) {
-			x.1.clone()
+			record.clone()
 		} else {
-			let fun = Rc::new(move |v| clone!(f; async move { f.apply(v).await }.boxed_local()));
+			let record = process_args(f).await;
-			fung.insert(path.clone(), (F::ARITY, fun.clone()));
+			fung.insert(path.clone(), record.clone());
-			fun
+			record
 		};
-		Self { args: vec![], arity: F::ARITY, path, fun }
+		Self { args: vec![], path, record }
 	}
-	pub fn arity(&self) -> u8 { self.arity }
+	pub fn arity(&self) -> u8 { self.record.argtyps.len() as u8 }
 }
 impl Atomic for Fun {
 	type Data = ();
@@ -74,11 +104,10 @@ impl OwnedAtom for Fun {
 	async fn call_ref(&self, arg: Expr) -> GExpr {
 		std::io::Write::flush(&mut std::io::stderr()).unwrap();
 		let new_args = self.args.iter().cloned().chain([arg]).collect_vec();
-		if new_args.len() == self.arity.into() {
+		if new_args.len() == self.record.argtyps.len() {
-			(self.fun)(new_args).await.to_expr()
+			(self.record.fun)(new_args).await.to_expr().await
 		} else {
-			Self { args: new_args, arity: self.arity, fun: self.fun.clone(), path: self.path.clone() }
+			Self { args: new_args, record: self.record.clone(), path: self.path.clone() }.to_expr().await
-				.to_expr()
 		}
 	}
 	async fn call(self, arg: Expr) -> GExpr { self.call_ref(arg).await }
@@ -89,11 +118,13 @@ impl OwnedAtom for Fun {
 	async fn deserialize(mut ctx: impl DeserializeCtx, args: Self::Refs) -> Self {
 		let sys = ctx.sys();
 		let path = Sym::from_api(ctx.decode().await, sys.i()).await;
-		let (arity, fun) = sys.get_or_default::<FunsCtx>().0.lock().await.get(&path).unwrap().clone();
+		let record = (sys.get::<FunsCtx>().0.lock().await.get(&path))
-		Self { args, arity, path, fun }
+			.expect("Function missing during deserialization")
+			.clone();
+		Self { args, path, record }
 	}
-	async fn print<'a>(&'a self, _: &'a (impl FmtCtx + ?Sized + 'a)) -> FmtUnit {
+	async fn print_atom<'a>(&'a self, _: &'a (impl FmtCtx + ?Sized + 'a)) -> FmtUnit {
-		format!("{}:{}/{}", self.path, self.args.len(), self.arity).into()
+		format!("{}:{}/{}", self.path, self.args.len(), self.arity()).into()
 	}
 }
 
@@ -104,13 +135,11 @@ impl OwnedAtom for Fun {
 #[derive(Clone)]
 pub struct Lambda {
 	args: Vec<Expr>,
-	arity: u8,
+	record: FunRecord,
-	fun: Rc<dyn FunCB>,
 }
 impl Lambda {
-	pub fn new<I, O, F: ExprFunc<I, O>>(f: F) -> Self {
+	pub async fn new<I, O, F: ExprFunc<I, O>>(f: F) -> Self {
-		let fun = Rc::new(move |v| clone!(f; async move { f.apply(v).await }.boxed_local()));
+		Self { args: vec![], record: process_args(f).await }
-		Self { args: vec![], arity: F::ARITY, fun }
 	}
 }
 impl Atomic for Lambda {
@@ -122,53 +151,59 @@ impl OwnedAtom for Lambda {
 	async fn val(&self) -> Cow<'_, Self::Data> { Cow::Owned(()) }
 	async fn call_ref(&self, arg: Expr) -> GExpr {
 		let new_args = self.args.iter().cloned().chain([arg]).collect_vec();
-		if new_args.len() == self.arity.into() {
+		if new_args.len() == self.record.argtyps.len() {
-			(self.fun)(new_args).await.to_expr()
+			(self.record.fun)(new_args).await.to_expr().await
 		} else {
-			Self { args: new_args, arity: self.arity, fun: self.fun.clone() }.to_expr()
+			Self { args: new_args, record: self.record.clone() }.to_expr().await
 		}
 	}
 	async fn call(self, arg: Expr) -> GExpr { self.call_ref(arg).await }
 }
 
 mod expr_func_derives {
+	use std::any::TypeId;
+	use std::sync::OnceLock;
+
 	use orchid_base::error::OrcRes;
 
 	use super::ExprFunc;
 	use crate::conv::{ToExpr, TryFromExpr};
-	use crate::func_atom::Expr;
+	use crate::func_atom::{ExecHandle, Expr};
 	use crate::gen_expr::GExpr;
 
 	macro_rules! expr_func_derive {
-    ($arity: tt, $($t:ident),*) => {
+    ($($t:ident),*) => {
       pastey::paste!{
         impl<
-          $($t: TryFromExpr, )*
+          $($t: TryFromExpr + 'static, )*
 					Out: ToExpr,
           Func: AsyncFn($($t,)*) -> Out + Clone + Send + Sync + 'static
         > ExprFunc<($($t,)*), Out> for Func {
-          const ARITY: u8 = $arity;
+          fn argtyps() -> &'static [TypeId] {
-          async fn apply(&self, v: Vec<Expr>) -> OrcRes<GExpr> {
+            static STORE: OnceLock<Vec<TypeId>> = OnceLock::new();
-            assert_eq!(v.len(), Self::ARITY.into(), "Arity mismatch");
+            &*STORE.get_or_init(|| vec![$(TypeId::of::<$t>()),*])
+          }
+          async fn apply<'a>(&self, _: ExecHandle<'a>, v: Vec<Expr>) -> OrcRes<GExpr> {
+            assert_eq!(v.len(), Self::argtyps().len(), "Arity mismatch");
             let [$([< $t:lower >],)*] = v.try_into().unwrap_or_else(|_| panic!("Checked above"));
-            Ok(self($($t::try_from_expr([< $t:lower >]).await?,)*).await.to_expr())
+            Ok(self($($t::try_from_expr([< $t:lower >]).await?,)*).await.to_expr().await)
           }
         }
       }
     };
   }
-	expr_func_derive!(1, A);
+	expr_func_derive!(A);
-	expr_func_derive!(2, A, B);
+	expr_func_derive!(A, B);
-	expr_func_derive!(3, A, B, C);
+	expr_func_derive!(A, B, C);
-	expr_func_derive!(4, A, B, C, D);
+	expr_func_derive!(A, B, C, D);
-	expr_func_derive!(5, A, B, C, D, E);
+	expr_func_derive!(A, B, C, D, E);
-	expr_func_derive!(6, A, B, C, D, E, F);
+	// expr_func_derive!(A, B, C, D, E, F);
-	expr_func_derive!(7, A, B, C, D, E, F, G);
+	// expr_func_derive!(A, B, C, D, E, F, G);
-	expr_func_derive!(8, A, B, C, D, E, F, G, H);
+	// expr_func_derive!(A, B, C, D, E, F, G, H);
-	expr_func_derive!(9, A, B, C, D, E, F, G, H, I);
+	// expr_func_derive!(A, B, C, D, E, F, G, H, I);
-	expr_func_derive!(10, A, B, C, D, E, F, G, H, I, J);
+	// expr_func_derive!(A, B, C, D, E, F, G, H, I, J);
-	expr_func_derive!(11, A, B, C, D, E, F, G, H, I, J, K);
+	// expr_func_derive!(A, B, C, D, E, F, G, H, I, J, K);
-	expr_func_derive!(12, A, B, C, D, E, F, G, H, I, J, K, L);
+	// expr_func_derive!(A, B, C, D, E, F, G, H, I, J, K, L);
-	expr_func_derive!(13, A, B, C, D, E, F, G, H, I, J, K, L, M);
+	// expr_func_derive!(A, B, C, D, E, F, G, H, I, J, K, L, M);
-	expr_func_derive!(14, A, B, C, D, E, F, G, H, I, J, K, L, M, N);
+	// expr_func_derive!(A, B, C, D, E, F, G, H, I, J, K, L, M, N);
 }

orchid-extension/src/gen_expr.rs

@@ -10,9 +10,7 @@ use orchid_base::{match_mapping, tl_cache};
 
 use crate::api;
 use crate::atom::{AtomFactory, ToAtom};
-use crate::conv::{ToExpr, TryFromExpr};
 use crate::expr::Expr;
-use crate::func_atom::Lambda;
 use crate::system::SysCtx;
 
 #[derive(Clone, Debug)]
@@ -35,6 +33,7 @@ impl GExpr {
 			}
 		}
 	}
+	pub fn at(self, pos: Pos) -> Self { GExpr { pos, kind: self.kind } }
 }
 impl Format for GExpr {
 	async fn print<'a>(&'a self, c: &'a (impl FmtCtx + ?Sized + 'a)) -> FmtUnit {
@@ -129,10 +128,3 @@ pub fn call(v: impl IntoIterator<Item = GExpr>) -> GExpr {
 pub fn bot(ev: impl IntoIterator<Item = OrcErr>) -> GExpr {
 	inherit(GExprKind::Bottom(OrcErrv::new(ev).unwrap()))
 }
-
-pub fn with<I: TryFromExpr, O: ToExpr>(
-	expr: GExpr,
-	cont: impl AsyncFn(I) -> O + Clone + Send + Sync + 'static,
-) -> GExpr {
-	call([lambda(0, [seq([arg(0), call([Lambda::new(cont).to_expr(), arg(0)])])]), expr])
-}

orchid-extension/src/lexer.rs

@@ -4,31 +4,33 @@ use std::ops::RangeInclusive;
 
 use futures::FutureExt;
 use futures::future::LocalBoxFuture;
-use orchid_base::error::{OrcErr, OrcRes, mk_err};
+use orchid_base::error::{OrcErrv, OrcRes, Reporter, mk_errv};
 use orchid_base::interner::{Interner, Tok};
 use orchid_base::location::{Pos, SrcRange};
 use orchid_base::name::Sym;
+use orchid_base::parse::ParseCtx;
 use orchid_base::reqnot::Requester;
 
 use crate::api;
+use crate::parser::PTokTree;
 use crate::system::SysCtx;
 use crate::tree::GenTokTree;
 
-pub async fn err_cascade(i: &Interner) -> OrcErr {
+pub async fn ekey_cascade(i: &Interner) -> Tok<String> {
-	mk_err(
+	i.i("An error cascading from a recursive call").await
-		i.i("An error cascading from a recursive call").await,
+}
-		"This error is a sentinel for the extension library.\
+pub async fn ekey_not_applicable(i: &Interner) -> Tok<String> {
-		it should not be emitted by the extension.",
+	i.i("Pseudo-error to communicate that the current branch in a dispatch doesn't apply").await
-		[Pos::None.into()],
+}
-	)
+const MSG_INTERNAL_ERROR: &str = "This error is a sentinel for the extension library.\
+	it should not be emitted by the extension.";
+
+pub async fn err_cascade(i: &Interner) -> OrcErrv {
+	mk_errv(ekey_cascade(i).await, MSG_INTERNAL_ERROR, [Pos::None])
 }
 
-pub async fn err_not_applicable(i: &Interner) -> OrcErr {
+pub async fn err_not_applicable(i: &Interner) -> OrcErrv {
-	mk_err(
+	mk_errv(ekey_not_applicable(i).await, MSG_INTERNAL_ERROR, [Pos::None])
-		i.i("Pseudo-error to communicate that the current branch in a dispatch doesn't apply").await,
-		&*err_cascade(i).await.message,
-		[Pos::None.into()],
-	)
 }
 
 pub struct LexContext<'a> {
@@ -36,16 +38,22 @@ pub struct LexContext<'a> {
 	pub text: &'a Tok<String>,
 	pub id: api::ParsId,
 	pub pos: u32,
-	pub src: Sym,
+	pub(crate) src: Sym,
+	pub(crate) rep: &'a Reporter,
 }
 impl<'a> LexContext<'a> {
-	pub async fn recurse(&self, tail: &'a str) -> OrcRes<(&'a str, GenTokTree)> {
+	pub fn src(&self) -> &Sym { &self.src }
+	/// This function returns [PTokTree] because it can never return
+	/// [orchid_base::tree::Token::NewExpr]. You can use
+	/// [crate::parser::p_tree2gen] to convert this to [crate::tree::GenTokTree]
+	/// for embedding in the return value.
+	pub async fn recurse(&self, tail: &'a str) -> OrcRes<(&'a str, PTokTree)> {
 		let start = self.pos(tail);
 		let Some(lx) = self.ctx.reqnot().request(api::SubLex { pos: start, id: self.id }).await else {
-			return Err(err_cascade(self.ctx.i()).await.into());
+			return Err(err_cascade(self.ctx.i()).await);
 		};
 		let tree =
-			GenTokTree::from_api(&lx.tree, &mut self.ctx.clone(), &mut (), &self.src, self.ctx.i()).await;
+			PTokTree::from_api(&lx.tree, &mut self.ctx.clone(), &mut (), &self.src, self.ctx.i()).await;
 		Ok((&self.text[lx.pos as usize..], tree))
 	}
 
@@ -57,8 +65,10 @@ impl<'a> LexContext<'a> {
 	pub fn pos_lt(&self, len: impl TryInto<u32, Error: fmt::Debug>, tail: &'a str) -> SrcRange {
 		SrcRange::new(self.pos(tail) - len.try_into().unwrap()..self.pos(tail), &self.src)
 	}
-
+}
-	pub fn i(&self) -> &Interner { self.ctx.i() }
+impl ParseCtx for LexContext<'_> {
+	fn i(&self) -> &Interner { self.ctx.i() }
+	fn rep(&self) -> &Reporter { self.rep }
 }
 
 pub trait Lexer: Send + Sync + Sized + Default + 'static {

orchid-extension/src/lib.rs

@@ -4,6 +4,7 @@ pub mod atom;
 pub mod atom_owned;
 pub mod atom_thin;
 pub mod conv;
+pub mod coroutine_exec;
 pub mod entrypoint;
 pub mod expr;
 pub mod func_atom;
@@ -12,6 +13,7 @@ pub mod lexer;
 pub mod msg;
 pub mod other_system;
 pub mod parser;
+pub mod reflection;
 pub mod system;
 pub mod system_ctor;
 pub mod tokio;

orchid-extension/src/parser.rs

@@ -2,29 +2,48 @@ use std::marker::PhantomData;
 
 use async_stream::stream;
 use futures::future::{LocalBoxFuture, join_all};
-use futures::{FutureExt, Stream, StreamExt, pin_mut};
+use futures::{FutureExt, Stream, StreamExt};
 use itertools::Itertools;
 use never::Never;
 use orchid_api::ResolveNames;
-use orchid_base::error::{OrcRes, Reporter};
+use orchid_base::error::{OrcErrv, OrcRes, Reporter};
 use orchid_base::id_store::IdStore;
 use orchid_base::interner::{Interner, Tok};
 use orchid_base::location::SrcRange;
+use orchid_base::match_mapping;
 use orchid_base::name::Sym;
 use orchid_base::parse::{Comment, ParseCtx, Snippet};
 use orchid_base::reqnot::{ReqHandlish, Requester};
 use orchid_base::tree::{TokTree, Token, ttv_into_api};
 
 use crate::api;
+use crate::conv::ToExpr;
 use crate::expr::Expr;
 use crate::gen_expr::GExpr;
 use crate::system::{SysCtx, SysCtxEntry};
-use crate::tree::GenTokTree;
+use crate::tree::{GenTok, GenTokTree};
 
 pub type PTok = Token<Expr, Never>;
 pub type PTokTree = TokTree<Expr, Never>;
 pub type PSnippet<'a> = Snippet<'a, Expr, Never>;
 
+pub fn p_tok2gen(tok: PTok) -> GenTok {
+	match_mapping!(tok, PTok => GenTok {
+		Comment(s), Name(n), BR, Handle(ex), Bottom(err),
+		LambdaHead(arg => Box::new(p_tree2gen(*arg))),
+		NS(n, arg => Box::new(p_tree2gen(*arg))),
+		S(p, body () p_v2gen),
+	} {
+		PTok::NewExpr(never) => match never {}
+	})
+}
+pub fn p_tree2gen(tree: PTokTree) -> GenTokTree {
+	TokTree { tok: p_tok2gen(tree.tok), sr: tree.sr }
+}
+pub fn p_v2gen(v: impl IntoIterator<Item = PTokTree>) -> Vec<GenTokTree> {
+	v.into_iter().map(p_tree2gen).collect_vec()
+}
+
 pub trait Parser: Send + Sync + Sized + Default + 'static {
 	const LINE_HEAD: &'static str;
 	fn parse<'a>(
@@ -93,6 +112,31 @@ pub struct ParsedLine {
 	pub kind: ParsedLineKind,
 }
 impl ParsedLine {
+	pub fn cnst<'a, R: ToExpr + 'static, F: AsyncFnOnce(ConstCtx) -> R + 'static>(
+		sr: &SrcRange,
+		comments: impl IntoIterator<Item = &'a Comment>,
+		exported: bool,
+		name: Tok<String>,
+		f: F,
+	) -> Self {
+		let cb = Box::new(|ctx| async move { f(ctx).await.to_expr().await }.boxed_local());
+		let kind = ParsedLineKind::Mem(ParsedMem { name, exported, kind: ParsedMemKind::Const(cb) });
+		let comments = comments.into_iter().cloned().collect();
+		ParsedLine { comments, sr: sr.clone(), kind }
+	}
+	pub fn module<'a>(
+		sr: &SrcRange,
+		comments: impl IntoIterator<Item = &'a Comment>,
+		exported: bool,
+		name: &Tok<String>,
+		use_prelude: bool,
+		lines: impl IntoIterator<Item = ParsedLine>,
+	) -> Self {
+		let mem_kind = ParsedMemKind::Mod { lines: lines.into_iter().collect(), use_prelude };
+		let line_kind = ParsedLineKind::Mem(ParsedMem { name: name.clone(), exported, kind: mem_kind });
+		let comments = comments.into_iter().cloned().collect();
+		ParsedLine { comments, sr: sr.clone(), kind: line_kind }
+	}
 	pub async fn into_api(self, ctx: SysCtx, hand: &dyn ReqHandlish) -> api::ParsedLine {
 		api::ParsedLine {
 			comments: self.comments.into_iter().map(|c| c.to_api()).collect(),
@@ -142,18 +186,6 @@ pub enum ParsedMemKind {
 	Mod { lines: Vec<ParsedLine>, use_prelude: bool },
 }
 
-impl ParsedMemKind {
-	pub fn cnst<F: AsyncFnOnce(ConstCtx) -> GExpr + 'static>(f: F) -> Self {
-		Self::Const(Box::new(|ctx| Box::pin(f(ctx))))
-	}
-	pub fn module(lines: impl IntoIterator<Item = ParsedLine>) -> Self {
-		Self::Mod { lines: lines.into_iter().collect(), use_prelude: true }
-	}
-	pub fn clean_module(lines: impl IntoIterator<Item = ParsedLine>) -> Self {
-		Self::Mod { lines: lines.into_iter().collect(), use_prelude: false }
-	}
-}
-
 /* TODO: how the macro runner uses the multi-stage loader
 
 Since the macro runner actually has to invoke the interpreter,
@@ -169,16 +201,18 @@ postparse / const load links up constants with every macro they can directly inv
 	the constants representing the keywords resolve to panic
 execute relies on these links detected in the extension to dispatch relevant macros
 */
-
+#[derive(Clone)]
 pub struct ConstCtx {
 	ctx: SysCtx,
 	constid: api::ParsedConstId,
 }
 impl ConstCtx {
-	pub fn names<'a>(
+	pub fn ctx(&self) -> &SysCtx { &self.ctx }
-		&'a self,
+	pub fn i(&self) -> &Interner { self.ctx.i() }
-		names: impl IntoIterator<Item = &'a Sym> + 'a,
+	pub fn names<'b>(
-	) -> impl Stream<Item = Option<Sym>> + 'a {
+		&'b self,
+		names: impl IntoIterator<Item = &'b Sym> + 'b,
+	) -> impl Stream<Item = OrcRes<Sym>> + 'b {
 		let resolve_names = ResolveNames {
 			constid: self.constid,
 			sys: self.ctx.sys_id(),
@@ -187,20 +221,14 @@ impl ConstCtx {
 		stream! {
 			for name_opt in self.ctx.reqnot().request(resolve_names).await {
 				yield match name_opt {
-					None => None,
+					Err(e) => Err(OrcErrv::from_api(&e, self.ctx.i()).await),
-					Some(name) => Some(Sym::from_api(name, self.ctx.i()).await)
+					Ok(name) => Ok(Sym::from_api(name, self.ctx.i()).await)
 				}
 			}
 		}
 	}
-	pub async fn names_n<const N: usize>(&self, names: [&Sym; N]) -> [Option<Sym>; N] {
+	pub async fn names_n<const N: usize>(&self, names: [&Sym; N]) -> [OrcRes<Sym>; N] {
-		let mut results = [const { None }; N];
+		self.names(names).collect::<Vec<_>>().await.try_into().expect("Lengths must match")
-		let names = self.names(names).enumerate().filter_map(|(i, n)| async move { Some((i, n?)) });
-		pin_mut!(names);
-		while let Some((i, name)) = names.next().await {
-			results[i] = Some(name);
-		}
-		results
 	}
 }
 

orchid-extension/src/reflection.rs

@@ -0,0 +1,163 @@
+use std::cell::OnceCell;
+use std::rc::Rc;
+
+use async_std::sync::Mutex;
+use futures::FutureExt;
+use memo_map::MemoMap;
+use orchid_base::interner::Tok;
+use orchid_base::name::{NameLike, VPath};
+use orchid_base::reqnot::Requester;
+
+use crate::api;
+use crate::system::{SysCtx, SysCtxEntry, WeakSysCtx};
+
+pub struct ReflMemData {
+	// None for inferred steps
+	public: OnceCell<bool>,
+	kind: ReflMemKind,
+}
+#[derive(Clone)]
+pub struct ReflMem(Rc<ReflMemData>);
+impl ReflMem {
+	pub fn kind(&self) -> ReflMemKind { self.0.kind.clone() }
+}
+
+#[derive(Clone)]
+pub enum ReflMemKind {
+	Const,
+	Mod(ReflMod),
+}
+
+pub struct ReflModData {
+	inferred: Mutex<bool>,
+	path: VPath,
+	ctx: WeakSysCtx,
+	members: MemoMap<Tok<String>, ReflMem>,
+}
+
+#[derive(Clone)]
+pub struct ReflMod(Rc<ReflModData>);
+impl ReflMod {
+	fn ctx(&self) -> SysCtx {
+		self.0.ctx.upgrade().expect("ReflectedModule accessed after context drop")
+	}
+	pub fn path(&self) -> &[Tok<String>] { &self.0.path[..] }
+	pub fn is_root(&self) -> bool { self.0.path.is_empty() }
+	async fn try_populate(&self) -> Result<(), api::LsModuleError> {
+		let ctx = self.ctx();
+		let path_tok = ctx.i().i(&self.0.path[..]).await;
+		let reply = match ctx.reqnot().request(api::LsModule(ctx.sys_id(), path_tok.to_api())).await {
+			Err(api::LsModuleError::TreeUnavailable) =>
+				panic!("Reflected tree accessed outside an interpreter call. This extension is faulty."),
+			Err(err) => return Err(err),
+			Ok(details) => details,
+		};
+		for (k, v) in reply.members {
+			let k = ctx.i().ex(k).await;
+			let mem = match self.0.members.get(&k) {
+				Some(mem) => mem,
+				None => {
+					let path = self.0.path.clone().name_with_suffix(k.clone()).to_sym(ctx.i()).await;
+					let kind = match v.kind {
+						api::MemberInfoKind::Constant => ReflMemKind::Const,
+						api::MemberInfoKind::Module =>
+							ReflMemKind::Mod(default_module(&ctx, VPath::new(path.segs()))),
+					};
+					self.0.members.get_or_insert(&k, || default_member(self.is_root(), kind))
+				},
+			};
+			let _ = mem.0.public.set(v.public);
+		}
+		Ok(())
+	}
+	pub async fn get_child(&self, key: &Tok<String>) -> Option<ReflMem> {
+		let inferred_g = self.0.inferred.lock().await;
+		if let Some(mem) = self.0.members.get(key) {
+			return Some(mem.clone());
+		}
+		if !*inferred_g {
+			return None;
+		}
+		match self.try_populate().await {
+			Err(api::LsModuleError::InvalidPath) =>
+				panic!("Path became invalid since module was created"),
+			Err(api::LsModuleError::IsConstant) =>
+				panic!("Path previously contained a module but now contains a constant"),
+			Err(api::LsModuleError::TreeUnavailable) => unreachable!(),
+			Ok(()) => (),
+		}
+		self.0.members.get(key).cloned()
+	}
+	pub async fn get_by_path(&self, path: &[Tok<String>]) -> Result<ReflMem, InvalidPathError> {
+		let ctx = self.ctx();
+		let (next, tail) = path.split_first().expect("Attempted to walk by empty path");
+		let inferred_g = self.0.inferred.lock().await;
+		if let Some(next) = self.0.members.get(next) {
+			return if tail.is_empty() {
+				Ok(next.clone())
+			} else {
+				match next.kind() {
+					ReflMemKind::Const => Err(InvalidPathError { keep_ancestry: true }),
+					ReflMemKind::Mod(m) => m.get_by_path(tail).boxed_local().await,
+				}
+			};
+		}
+		if !*inferred_g {
+			return Err(InvalidPathError { keep_ancestry: true });
+		}
+		let candidate = default_module(&ctx, self.0.path.clone().suffix([next.clone()]));
+		if tail.is_empty() {
+			return match candidate.try_populate().await {
+				Ok(()) => {
+					let tgt_mem = default_member(self.is_root(), ReflMemKind::Mod(candidate));
+					self.0.members.insert(next.clone(), tgt_mem.clone());
+					Ok(tgt_mem)
+				},
+				Err(api::LsModuleError::InvalidPath) => Err(InvalidPathError { keep_ancestry: false }),
+				Err(api::LsModuleError::IsConstant) => {
+					let const_mem = default_member(self.is_root(), ReflMemKind::Const);
+					self.0.members.insert(next.clone(), const_mem);
+					Err(InvalidPathError { keep_ancestry: true })
+				},
+				Err(api::LsModuleError::TreeUnavailable) => unreachable!(),
+			};
+		}
+		match candidate.get_by_path(tail).boxed_local().await {
+			e @ Err(InvalidPathError { keep_ancestry: false }) => e,
+			res @ Err(InvalidPathError { keep_ancestry: true }) | res @ Ok(_) => {
+				let tgt_mem = default_member(self.is_root(), ReflMemKind::Mod(candidate));
+				self.0.members.insert(next.clone(), tgt_mem);
+				res
+			},
+		}
+	}
+}
+
+struct ReflRoot(ReflMod);
+impl SysCtxEntry for ReflRoot {}
+
+pub struct InvalidPathError {
+	keep_ancestry: bool,
+}
+
+fn default_module(ctx: &SysCtx, path: VPath) -> ReflMod {
+	ReflMod(Rc::new(ReflModData {
+		ctx: ctx.downgrade(),
+		inferred: Mutex::new(true),
+		path,
+		members: MemoMap::new(),
+	}))
+}
+
+fn default_member(is_root: bool, kind: ReflMemKind) -> ReflMem {
+	ReflMem(Rc::new(ReflMemData {
+		public: if is_root { true.into() } else { OnceCell::new() },
+		kind,
+	}))
+}
+
+fn get_root(ctx: &SysCtx) -> &ReflRoot {
+	ctx.get_or_insert(|| ReflRoot(default_module(ctx, VPath::new([]))))
+}
+
+pub fn refl(ctx: &SysCtx) -> ReflMod { get_root(ctx).0.clone() }

orchid-extension/src/system.rs

@@ -3,7 +3,7 @@ use std::fmt;
 use std::future::Future;
 use std::num::NonZero;
 use std::pin::Pin;
-use std::rc::Rc;
+use std::rc::{Rc, Weak};
 
 use futures::future::LocalBoxFuture;
 use memo_map::MemoMap;
@@ -36,7 +36,7 @@ pub trait DynSystemCard: Send + Sync + 'static {
 	fn name(&self) -> &'static str;
 	/// Atoms explicitly defined by the system card. Do not rely on this for
 	/// querying atoms as it doesn't include the general atom types
-	fn atoms(&self) -> BoxedIter<Option<Box<dyn AtomDynfo>>>;
+	fn atoms(&'_ self) -> BoxedIter<'_, Option<Box<dyn AtomDynfo>>>;
 }
 
 /// Atoms supported by this package which may appear in all extensions.
@@ -77,7 +77,9 @@ pub async fn resolv_atom(
 
 impl<T: SystemCard> DynSystemCard for T {
 	fn name(&self) -> &'static str { T::Ctor::NAME }
-	fn atoms(&self) -> BoxedIter<Option<Box<dyn AtomDynfo>>> { Box::new(Self::atoms().into_iter()) }
+	fn atoms(&'_ self) -> BoxedIter<'_, Option<Box<dyn AtomDynfo>>> {
+		Box::new(Self::atoms().into_iter())
+	}
 }
 
 /// System as defined by author
@@ -91,7 +93,7 @@ pub trait System: Send + Sync + SystemCard + 'static {
 
 pub trait DynSystem: Send + Sync + DynSystemCard + 'static {
 	fn dyn_prelude<'a>(&'a self, i: &'a Interner) -> LocalBoxFuture<'a, Vec<Sym>>;
-	fn dyn_env(&self) -> Vec<GenMember>;
+	fn dyn_env(&'_ self) -> Vec<GenMember>;
 	fn dyn_lexers(&self) -> Vec<LexerObj>;
 	fn dyn_parsers(&self) -> Vec<ParserObj>;
 	fn dyn_request<'a>(&self, hand: ExtReq<'a>, req: Vec<u8>) -> LocalBoxFuture<'a, Receipt<'a>>;
@@ -102,7 +104,7 @@ impl<T: System> DynSystem for T {
 	fn dyn_prelude<'a>(&'a self, i: &'a Interner) -> LocalBoxFuture<'a, Vec<Sym>> {
 		Box::pin(Self::prelude(i))
 	}
-	fn dyn_env(&self) -> Vec<GenMember> { Self::env() }
+	fn dyn_env(&'_ self) -> Vec<GenMember> { Self::env() }
 	fn dyn_lexers(&self) -> Vec<LexerObj> { Self::lexers() }
 	fn dyn_parsers(&self) -> Vec<ParserObj> { Self::parsers() }
 	fn dyn_request<'a>(&self, hand: ExtReq<'a>, req: Vec<u8>) -> LocalBoxFuture<'a, Receipt<'a>> {
@@ -132,7 +134,13 @@ where A: AtomicFeatures {
 	}
 	let val = dynfo.decode(AtomCtx(data, foreign.atom.drop, ctx)).await;
 	let value = *val.downcast::<A::Data>().expect("atom decode returned wrong type");
-	Ok(TypAtom { value, data: foreign })
+	Ok(TypAtom { value, untyped: foreign })
+}
+
+#[derive(Clone)]
+pub struct WeakSysCtx(Weak<MemoMap<TypeId, Box<dyn Any>>>);
+impl WeakSysCtx {
+	pub fn upgrade(&self) -> Option<SysCtx> { Some(SysCtx(self.0.upgrade()?)) }
 }
 
 #[derive(Clone)]
@@ -150,6 +158,7 @@ impl SysCtx {
 		this.add(id).add(i).add(reqnot).add(spawner).add(logger).add(cted);
 		this
 	}
+	pub fn downgrade(&self) -> WeakSysCtx { WeakSysCtx(Rc::downgrade(&self.0)) }
 	pub fn add<T: SysCtxEntry>(&self, t: T) -> &Self {
 		assert!(self.0.insert(TypeId::of::<T>(), Box::new(t)), "Key already exists");
 		self

orchid-extension/src/tree.rs

@@ -65,17 +65,24 @@ impl TokenVariant<api::ExprTicket> for Expr {
 	}
 }
 
-pub fn x_tok(x: impl ToExpr) -> GenTok { GenTok::NewExpr(x.to_expr()) }
+pub async fn x_tok(x: impl ToExpr) -> GenTok { GenTok::NewExpr(x.to_expr().await) }
-pub fn ref_tok(path: Sym) -> GenTok { GenTok::NewExpr(sym_ref(path)) }
+pub async fn ref_tok(path: Sym) -> GenTok { GenTok::NewExpr(sym_ref(path)) }
 
-pub fn cnst(public: bool, name: &str, value: impl ToExpr) -> Vec<GenMember> {
+pub fn lazy(
+	public: bool,
+	name: &str,
+	cb: impl AsyncFnOnce(Sym, SysCtx) -> MemKind + Clone + 'static,
+) -> Vec<GenMember> {
 	vec![GenMember {
 		name: name.to_string(),
-		kind: MemKind::Const(value.to_expr()),
+		kind: MemKind::Lazy(LazyMemberFactory::new(cb)),
 		comments: vec![],
 		public,
 	}]
 }
+pub fn cnst(public: bool, name: &str, value: impl ToExpr + Clone + 'static) -> Vec<GenMember> {
+	lazy(public, name, async |_, _| MemKind::Const(value.to_expr().await))
+}
 pub fn module(
 	public: bool,
 	name: &str,
@@ -89,20 +96,19 @@ pub fn root_mod(name: &str, mems: impl IntoIterator<Item = Vec<GenMember>>) -> (
 	(name.to_string(), kind)
 }
 pub fn fun<I, O>(public: bool, name: &str, xf: impl ExprFunc<I, O>) -> Vec<GenMember> {
-	let fac = LazyMemberFactory::new(move |sym, ctx| async {
+	let fac =
-		return MemKind::Const(build_lambdas(Fun::new(sym, ctx, xf).await, 0));
+		LazyMemberFactory::new(move |sym, ctx| async {
-		fn build_lambdas(fun: Fun, i: u64) -> GExpr {
+			return MemKind::Const(build_lambdas(Fun::new(sym, ctx, xf).await, 0).await);
-			if i < fun.arity().into() {
+			async fn build_lambdas(fun: Fun, i: u64) -> GExpr {
-				return lambda(i, [build_lambdas(fun, i + 1)]);
+				if i < fun.arity().into() {
+					return lambda(i, [build_lambdas(fun, i + 1).boxed_local().await]);
+				}
+				let arity = fun.arity();
+				seq((0..arity).map(|i| arg(i as u64)).chain([call(
+					[fun.to_expr().await].into_iter().chain((0..arity).map(|i| arg(i as u64))),
+				)]))
 			}
-			let arity = fun.arity();
+		});
-			seq(
-				(0..arity)
-					.map(|i| arg(i as u64))
-					.chain([call([fun.to_expr()].into_iter().chain((0..arity).map(|i| arg(i as u64))))]),
-			)
-		}
-	});
 	vec![GenMember { name: name.to_string(), kind: MemKind::Lazy(fac), public, comments: vec![] }]
 }
 pub fn prefix(path: &str, items: impl IntoIterator<Item = Vec<GenMember>>) -> Vec<GenMember> {

orchid-host/src/atom.rs

@@ -58,10 +58,10 @@ impl AtomHand {
 		};
 		if let Some(id) = drop {
 			let mut owned_g = ctx.owned_atoms.write().await;
-			if let Some(data) = owned_g.get(&id) {
+			if let Some(data) = owned_g.get(&id)
-				if let Some(atom) = data.upgrade() {
+				&& let Some(atom) = data.upgrade()
-					return atom;
+			{
-				}
+				return atom;
 			}
 			let new = create().await;
 			owned_g.insert(id, new.downgrade());

orchid-host/src/dealias.rs

@@ -1,6 +1,6 @@
 use hashbrown::HashSet;
 use itertools::Itertools;
-use orchid_base::error::{OrcErr, OrcRes, Reporter, mk_err, mk_errv};
+use orchid_base::error::{OrcErrv, OrcRes, Reporter, mk_errv};
 use orchid_base::interner::{Interner, Tok};
 use orchid_base::location::Pos;
 use orchid_base::name::VName;
@@ -16,7 +16,7 @@ pub enum AbsPathError {
 	RootPath,
 }
 impl AbsPathError {
-	pub async fn err_obj(self, i: &Interner, pos: Pos, path: &str) -> OrcErr {
+	pub async fn err_obj(self, i: &Interner, pos: Pos, path: &str) -> OrcErrv {
 		let (descr, msg) = match self {
 			AbsPathError::RootPath => (
 				i.i("Path ends on root module").await,
@@ -30,7 +30,7 @@ impl AbsPathError {
 				format!("{path} is leading outside the root."),
 			),
 		};
-		mk_err(descr, msg, [pos.into()])
+		mk_errv(descr, msg, [pos])
 	}
 }
 

orchid-host/src/extension.rs

@@ -206,12 +206,16 @@ impl Extension {
 											let sys = weak_sys.upgrade().expect("ResolveNames after sys drop");
 											sys.name_resolver(*constid).await
 										};
-										let mut responses = vec![const { None }; names.len()];
+										let responses = stream! {
-										for (i, name) in names.iter().enumerate() {
+											for name in names {
-											if let Some(abs) = resolver(&ctx.i.ex(*name).await[..]).await {
+												yield match resolver(&ctx.i.ex(*name).await[..]).await {
-												responses[i] = Some(abs.to_sym(&ctx.i).await.to_api())
+													Ok(abs) => Ok(abs.to_sym(&ctx.i).await.to_api()),
+													Err(e) => Err(e.to_api()),
+												}
 											}
 										}
+										.collect()
+										.await;
 										hand.handle(rn, &responses).await
 									},
 									api::ExtHostReq::ExtAtomPrint(ref eap @ api::ExtAtomPrint(ref atom)) => {

orchid-host/src/system.rs

@@ -10,11 +10,12 @@ use hashbrown::HashMap;
 use itertools::Itertools;
 use memo_map::MemoMap;
 use orchid_base::char_filter::char_filter_match;
-use orchid_base::error::{OrcErrv, OrcRes};
+use orchid_base::error::{OrcErrv, OrcRes, mk_errv_floating};
 use orchid_base::format::{FmtCtx, FmtUnit, Format};
 use orchid_base::interner::{Interner, Tok};
+use orchid_base::iter_utils::IteratorPrint;
 use orchid_base::location::SrcRange;
-use orchid_base::name::{NameLike, Sym, VName};
+use orchid_base::name::{NameLike, Sym, VName, VPath};
 use orchid_base::parse::Comment;
 use orchid_base::reqnot::{ReqNot, Requester};
 use orchid_base::tree::ttv_from_api;
@@ -23,7 +24,7 @@ use substack::{Stackframe, Substack};
 
 use crate::api;
 use crate::ctx::Ctx;
-use crate::dealias::{absolute_path, walk};
+use crate::dealias::walk;
 use crate::expr::{ExprParseCtx, ExprWillPanic};
 use crate::expr_store::ExprStore;
 use crate::extension::{Extension, WeakExtension};
@@ -202,16 +203,39 @@ impl System {
 	pub(crate) async fn name_resolver(
 		&self,
 		orig: api::ParsedConstId,
-	) -> impl AsyncFnMut(&[Tok<String>]) -> Option<VName> + use<> {
+	) -> impl AsyncFnMut(&[Tok<String>]) -> OrcRes<VName> + use<> {
 		let root = self.0.ctx.root.read().await.upgrade().expect("find_names when root not in context");
 		let orig = self.0.const_paths.get(&orig).expect("origin for find_names invalid").clone();
 		let ctx = self.0.ctx.clone();
 		async move |rel| {
 			let cwd = orig.split_last_seg().1;
-			let abs = absolute_path(cwd, rel, &ctx.i).await.ok()?;
 			let root_data = &mut *root.0.write().await;
 			let walk_ctx = &mut (ctx.clone(), &mut root_data.consts);
-			walk(&root_data.root, false, abs.iter(), walk_ctx).await.is_ok().then_some(abs)
+			let cmod = (walk(&root_data.root, false, cwd.iter().cloned(), walk_ctx).await)
+				.expect("the parent module of a constant should exist");
+			let (selector, tail) = rel.split_first().expect("Names cannot be empty");
+			if cmod.members.get(selector).is_some() {
+				return Ok(VName::new(cwd.iter().chain(rel).cloned()).unwrap());
+			}
+			match cmod.imports.get(selector) {
+				Some(Ok(dest)) => return Ok(dest.target.to_vname().suffix(tail.iter().cloned())),
+				Some(Err(dests)) =>
+					return Err(mk_errv_floating(
+						ctx.i.i("Ambiguous name").await,
+						format!(
+							"{selector} could refer to {}",
+							dests.iter().map(|ri| &ri.target).display("or")
+						),
+					)),
+				None => (),
+			}
+			if tail.is_empty() {
+				return Ok(VPath::new(cwd.iter().cloned()).name_with_suffix(selector.clone()));
+			}
+			Err(mk_errv_floating(
+				ctx.i.i("Invalid name").await,
+				format!("{selector} doesn't refer to a module"),
+			))
 		}
 	}
 }

orchid-host/src/tree.rs

@@ -2,6 +2,7 @@
 //! once
 use std::cell::RefCell;
 use std::rc::{Rc, Weak};
+use std::slice;
 
 use async_once_cell::OnceCell;
 use async_std::sync::RwLock;
@@ -10,7 +11,7 @@ use hashbrown::HashMap;
 use hashbrown::hash_map::Entry;
 use itertools::Itertools;
 use orchid_base::clone;
-use orchid_base::error::{OrcRes, Reporter, mk_err, mk_errv};
+use orchid_base::error::{OrcRes, Reporter, mk_errv};
 use orchid_base::interner::Tok;
 use orchid_base::location::{CodeGenInfo, Pos};
 use orchid_base::name::{NameLike, Sym, VPath};
@@ -152,8 +153,8 @@ impl<'a> TreeFromApiCtx<'a> {
 
 #[derive(Clone, Debug, PartialEq, Eq)]
 pub struct ResolvedImport {
-	target: Sym,
+	pub target: Sym,
-	pos: Pos,
+	pub pos: Pos,
 }
 
 #[derive(Clone, Default)]
@@ -258,18 +259,16 @@ impl Module {
 				match abs_path_res {
 					Err(e) => ctx.rep.report(e.err_obj(&ctx.ctx.i, sr.pos(), &import.to_string()).await),
 					Ok(abs_path) => {
-						imports.insert(
+						let target = abs_path.to_sym(&ctx.ctx.i).await;
-							key,
+						imports.insert(key, Ok(ResolvedImport { target, pos: sr.pos() }));
-							Ok(ResolvedImport { target: abs_path.to_sym(&ctx.ctx.i).await, pos: sr.pos() }),
-						);
 					},
 				}
 			} else {
 				for item in values {
-					ctx.rep.report(mk_err(
+					ctx.rep.report(mk_errv(
 						conflicting_imports_msg.clone(),
 						format!("{key} is imported multiple times from different modules"),
-						[item.sr.pos().into()],
+						[item.sr.pos()],
 					));
 				}
 			}
@@ -294,11 +293,12 @@ impl Module {
 		let self_referential_msg = ctx.ctx.i.i("Self-referential import").await;
 		for (key, value) in imports.iter() {
 			let Ok(import) = value else { continue };
-			if import.target.strip_prefix(&path[..]).is_some_and(|t| t.starts_with(&[key.clone()])) {
+			if import.target.strip_prefix(&path[..]).is_some_and(|t| t.starts_with(slice::from_ref(key)))
-				ctx.rep.report(mk_err(
+			{
+				ctx.rep.report(mk_errv(
 					self_referential_msg.clone(),
 					format!("import {} points to itself or a path within itself", &import.target),
-					[import.pos.clone().into()],
+					[import.pos.clone()],
 				));
 			}
 		}

orchid-std/src/lib.rs

@@ -4,3 +4,6 @@ mod std;
 pub use std::number::num_atom::{Float, HomoArray, Int, Num};
 pub use std::std_system::StdSystem;
 pub use std::string::str_atom::OrcString;
+
+pub use macros::macro_system::MacroSystem;
+pub use macros::mactree::{MacTok, MacTree};

orchid-std/src/macros/instantiate_tpl.rs

@@ -1,14 +1,11 @@
 use std::borrow::Cow;
-use std::pin::Pin;
 
-use futures::AsyncWrite;
 use never::Never;
 use orchid_extension::atom::{Atomic, TypAtom};
-use orchid_extension::atom_owned::{DeserializeCtx, OwnedAtom, OwnedVariant, get_own_instance};
+use orchid_extension::atom_owned::{OwnedAtom, OwnedVariant, own};
 use orchid_extension::conv::{ToExpr, TryFromExpr};
 use orchid_extension::expr::Expr;
 use orchid_extension::gen_expr::GExpr;
-use orchid_extension::system::SysCtx;
 
 use crate::macros::mactree::{MacTok, MacTree, map_mactree};
 
@@ -24,16 +21,7 @@ impl Atomic for InstantiateTplCall {
 }
 impl OwnedAtom for InstantiateTplCall {
 	async fn val(&self) -> Cow<'_, Self::Data> { Cow::Owned(()) }
-	/// TODO: get serialization done for mactree
+	type Refs = Never;
-	type Refs = Vec<Expr>;
-	async fn serialize(
-		&self,
-		ctx: SysCtx,
-		write: Pin<&mut (impl AsyncWrite + ?Sized)>,
-	) -> Self::Refs {
-		todo!()
-	}
-	async fn deserialize(ctx: impl DeserializeCtx, refs: Self::Refs) -> Self { todo!() }
 	// Technically must be supported but shouldn't actually ever be called
 	async fn call_ref(&self, arg: Expr) -> GExpr {
 		eprintln!(
@@ -43,20 +31,19 @@ impl OwnedAtom for InstantiateTplCall {
 		self.clone().call(arg).await
 	}
 	async fn call(mut self, arg: Expr) -> GExpr {
-		let arg = match TypAtom::try_from_expr(arg).await {
+		match TypAtom::<MacTree>::try_from_expr(arg).await {
-			Err(e) => return Err::<Never, _>(e).to_expr(),
+			Err(e) => return Err::<Never, _>(e).to_expr().await,
-			Ok(t) => get_own_instance(t).await,
+			Ok(t) => self.argv.push(own(t).await),
 		};
-		self.argv.push(arg);
 		if self.argv.len() < self.argc {
-			return self.to_expr();
+			return self.to_expr().await;
 		}
 		let mut args = self.argv.into_iter();
-		map_mactree(&self.tpl, &mut false, &mut |tpl| match &*tpl.tok {
+		let ret = map_mactree(&self.tpl, &mut false, &mut |mt| match mt.tok() {
-			MacTok::Ph(_) => panic!("instantiate_tpl received a placeholder"),
+			MacTok::Slot => Some(args.next().expect("Not enough arguments to fill all slots")),
-			MacTok::Slot => Some(args.next().expect("Not enough arguments to fill all slots!")),
 			_ => None,
-		})
+		});
-		.to_expr()
+		assert!(args.next().is_none(), "Too many arguments for all slots");
+		ret.to_expr().await
 	}
 }

orchid-std/src/macros/let_line.rs

@@ -1,17 +1,20 @@
+use std::pin::pin;
+
 use async_std::stream;
-use async_stream::stream;
 use futures::{FutureExt, StreamExt};
 use hashbrown::HashMap;
 use itertools::Itertools;
-use orchid_base::error::OrcRes;
+use orchid_api::Paren;
+use orchid_base::error::{OrcRes, Reporter};
+use orchid_base::name::Sym;
 use orchid_base::parse::{
 	Comment, ParseCtx, Parsed, Snippet, expect_tok, token_errv, try_pop_no_fluff,
 };
-use orchid_extension::parser::{
+use orchid_base::sym;
-	PSnippet, PTok, PTokTree, ParsCtx, ParsedLine, ParsedLineKind, ParsedMem, ParsedMemKind, Parser,
+use orchid_extension::gen_expr::{atom, call, sym_ref};
-};
+use orchid_extension::parser::{ConstCtx, PSnippet, PTok, PTokTree, ParsCtx, ParsedLine, Parser};
 
-use crate::macros::mactree::{MacTok, MacTree, map_mactree_v};
+use crate::macros::mactree::{MacTok, MacTree, glossary_v, map_mactree_v};
 
 #[derive(Default)]
 pub struct LetLine;
@@ -23,6 +26,7 @@ impl Parser for LetLine {
 		comments: Vec<Comment>,
 		line: PSnippet<'a>,
 	) -> OrcRes<Vec<ParsedLine>> {
+		let sr = line.sr();
 		let Parsed { output: name_tok, tail } = try_pop_no_fluff(&ctx, line).await?;
 		let Some(name) = name_tok.as_name() else {
 			let err = token_errv(&ctx, name_tok, "Constant must have a name", |t| {
@@ -31,44 +35,41 @@ impl Parser for LetLine {
 			return Err(err.await);
 		};
 		let Parsed { tail, .. } = expect_tok(&ctx, tail, ctx.i().i("=").await).await?;
-		let mut names = HashMap::new();
 		let aliased = parse_tokv(tail, &ctx).await;
-		map_mactree_v(&aliased, &mut false, &mut |tpl| {
+		Ok(vec![ParsedLine::cnst(&line.sr(), &comments, exported, name, async move |ctx| {
-			if let MacTok::Name(n) = &*tpl.tok {
+			let rep = Reporter::new();
-				names.insert(n.clone(), n.clone());
+			let dealiased = dealias_mac_v(aliased, &ctx, &rep).await;
+			let macro_input = MacTok::S(Paren::Round, dealiased).at(sr.pos());
+			if let Some(e) = rep.errv() {
+				return Err(e);
 			}
-			None
+			Ok(call([
-		});
+				sym_ref(sym!(macros::lower; ctx.i()).await),
-		Ok(vec![ParsedLine {
+				call([sym_ref(sym!(macros::resolve; ctx.i()).await), atom(macro_input)]),
-			comments,
+			]))
-			sr: line.sr(),
+		})])
-			kind: ParsedLineKind::Mem(ParsedMem {
-				exported,
-				name,
-				kind: ParsedMemKind::cnst(async move |ctx| {
-					let keys = names.keys().cloned().collect_vec();
-					let names_mut = &mut names;
-					stream! {
-						for await (canonical, local) in ctx.names(&keys).zip(stream::from_iter(&keys)) {
-							if let Some(name) = canonical {
-								*names_mut.get_mut(local).expect("Queried specifically the keys of this map") = name
-							}
-						}
-					}
-					.collect::<()>()
-					.await;
-					let macro_input = map_mactree_v(&aliased, &mut false, &mut |tree| match &*tree.tok {
-						MacTok::Name(n) => names.get(n).map(|new_n| MacTok::Name(new_n.clone()).at(tree.pos())),
-						_ => None,
-					});
-					todo!("Run macros then convert this into an expr")
-				}),
-			}),
-		}])
 	}
 }
 
-async fn parse_tokv(line: PSnippet<'_>, ctx: &ParsCtx<'_>) -> Vec<MacTree> {
+pub async fn dealias_mac_v(aliased: Vec<MacTree>, ctx: &ConstCtx, rep: &Reporter) -> Vec<MacTree> {
+	let keys = glossary_v(&aliased).collect_vec();
+	let mut names: HashMap<_, _> = HashMap::new();
+	let mut stream = pin!(ctx.names(&keys).zip(stream::from_iter(&keys)));
+	while let Some((canonical, local)) = stream.next().await {
+		match canonical {
+			Err(e) => rep.report(e),
+			Ok(name) => {
+				names.insert(local.clone(), name);
+			},
+		}
+	}
+	map_mactree_v(&aliased, &mut false, &mut |tree| match &*tree.tok {
+		MacTok::Name(n) => names.get(n).map(|new_n| MacTok::Name(new_n.clone()).at(tree.pos())),
+		_ => None,
+	})
+}
+
+pub async fn parse_tokv(line: PSnippet<'_>, ctx: &impl ParseCtx) -> Vec<MacTree> {
 	if let Some((idx, arg)) = line.iter().enumerate().find_map(|(i, x)| Some((i, x.as_lambda()?))) {
 		let (head, lambda) = line.split_at(idx as u32);
 		let (_, body) = lambda.pop_front().unwrap();
@@ -89,15 +90,15 @@ async fn parse_tokv(line: PSnippet<'_>, ctx: &ParsCtx<'_>) -> Vec<MacTree> {
 	}
 }
 
-async fn parse_tokv_no_lambdas(line: &[PTokTree], ctx: &ParsCtx<'_>) -> Vec<MacTree> {
+async fn parse_tokv_no_lambdas(line: &[PTokTree], ctx: &impl ParseCtx) -> Vec<MacTree> {
 	stream::from_iter(line).filter_map(|tt| parse_tok(tt, ctx)).collect().await
 }
 
-async fn parse_tok(tree: &PTokTree, ctx: &ParsCtx<'_>) -> Option<MacTree> {
+pub async fn parse_tok(tree: &PTokTree, ctx: &impl ParseCtx) -> Option<MacTree> {
 	let tok = match &tree.tok {
 		PTok::Bottom(errv) => MacTok::Bottom(errv.clone()),
 		PTok::BR | PTok::Comment(_) => return None,
-		PTok::Name(n) => MacTok::Name(ctx.module().suffix([n.clone()], ctx.i()).await),
+		PTok::Name(n) => MacTok::Name(Sym::new([n.clone()], ctx.i()).await.unwrap()),
 		PTok::NS(..) => match tree.as_multiname() {
 			Ok(mn) => MacTok::Name(mn.to_sym(ctx.i()).await),
 			Err(nested) => {

orchid-std/src/macros/macro_lib.rs

@@ -1,20 +1,83 @@
+use hashbrown::HashMap;
+use itertools::Itertools;
+use orchid_base::error::Reporter;
+use orchid_base::sym;
 use orchid_extension::atom::TypAtom;
-use orchid_extension::atom_owned::get_own_instance;
+use orchid_extension::atom_owned::own;
+use orchid_extension::conv::ToExpr;
+use orchid_extension::coroutine_exec::exec;
+use orchid_extension::gen_expr::{atom, call, sym_ref};
+use orchid_extension::reflection::{ReflMemKind, refl};
 use orchid_extension::tree::{GenMember, comments, fun, prefix};
+use substack::Substack;
 
 use crate::Int;
 use crate::macros::instantiate_tpl::InstantiateTplCall;
-use crate::macros::mactree::MacTree;
+use crate::macros::macro_line::{Macro, Matcher};
+use crate::macros::mactree::{LowerCtx, MacTree};
+use crate::macros::recur_state::RecurState;
+use crate::macros::resolve::{ResolveCtx, resolve};
 
 pub fn gen_macro_lib() -> Vec<GenMember> {
-	prefix("macros", [comments(
+	prefix("macros", [
-		["This is an internal function, you can't obtain a value of its argument type.", "hidden"],
+		comments(
-		fun(true, "instantiate_tpl", |tpl: TypAtom<MacTree>, right: Int| async move {
+			["This is an internal function, you can't obtain a value of its argument type.", "hidden"],
-			InstantiateTplCall {
+			fun(true, "instantiate_tpl", |tpl: TypAtom<MacTree>, right: Int| async move {
-				tpl: get_own_instance(tpl).await,
+				InstantiateTplCall {
-				argc: right.0.try_into().unwrap(),
+					tpl: own(tpl).await,
-				argv: Vec::new(),
+					argc: right.0.try_into().unwrap(),
-			}
+					argv: Vec::new(),
+				}
+			}),
+		),
+		fun(true, "resolve", |tpl: TypAtom<MacTree>| async move {
+			call([
+				sym_ref(sym!(macro::resolve_recur; tpl.untyped.ctx().i()).await),
+				atom(RecurState::Bottom),
+				tpl.untyped.ex().to_expr().await,
+			])
 		}),
-	)])
+		fun(true, "lower", |tpl: TypAtom<MacTree>| async move {
+			let ctx = LowerCtx { sys: tpl.untyped.ctx().clone(), rep: &Reporter::new() };
+			let res = own(tpl).await.lower(ctx, Substack::Bottom).await;
+			if let Some(e) = Reporter::new().errv() { Err(e) } else { Ok(res) }
+		}),
+		fun(true, "resolve_recur", |state: TypAtom<RecurState>, tpl: TypAtom<MacTree>| async move {
+			exec(async move |mut h| {
+				let ctx = tpl.ctx().clone();
+				let root = refl(&ctx);
+				let tpl = own(tpl.clone()).await;
+				let mut macros = HashMap::new();
+				for n in tpl.glossary() {
+					if let Ok(ReflMemKind::Const) = root.get_by_path(n).await.map(|m| m.kind()) {
+						let Ok(mac) = h.exec::<TypAtom<Macro>>(sym_ref(n.clone())).await else { continue };
+						let mac = own(mac).await;
+						macros.entry(mac.0.own_kws[0].clone()).or_insert(mac);
+					}
+				}
+				let mut named = HashMap::new();
+				let mut priod = Vec::new();
+				for (_, mac) in macros.iter() {
+					for rule in mac.0.rules.iter() {
+						if rule.glossary.is_subset(tpl.glossary()) {
+							match &rule.pattern {
+								Matcher::Named(m) =>
+									named.entry(m.head()).or_insert(Vec::new()).push((m, mac, rule)),
+								Matcher::Priod(p) => priod.push((mac.0.prio, (p, mac, rule))),
+							}
+						}
+					}
+				}
+				let priod = priod.into_iter().sorted_unstable_by_key(|(p, _)| *p).map(|(_, r)| r).collect();
+				let mut rctx = ResolveCtx { h, recur: own(state).await, ctx: ctx.clone(), named, priod };
+				let resolve_res = resolve(&mut rctx, &tpl).await;
+				std::mem::drop(rctx);
+				match resolve_res {
+					Some(out_tree) => out_tree.to_expr().await,
+					None => tpl.to_expr().await,
+				}
+			})
+			.await
+		}),
+	])
 }

orchid-std/src/macros/macro_line.rs

@@ -0,0 +1,230 @@
+use std::borrow::Cow;
+use std::cell::RefCell;
+use std::rc::Rc;
+
+use async_once_cell::OnceCell;
+use async_std::stream;
+use futures::StreamExt;
+use hashbrown::{HashMap, HashSet};
+use itertools::Itertools;
+use never::Never;
+use orchid_api::Paren;
+use orchid_base::error::{OrcRes, Reporter, mk_errv};
+use orchid_base::interner::Tok;
+use orchid_base::location::Pos;
+use orchid_base::name::Sym;
+use orchid_base::parse::{
+	Comment, ParseCtx, Parsed, Snippet, expect_end, expect_tok, line_items, token_errv,
+	try_pop_no_fluff,
+};
+use orchid_base::tree::Token;
+use orchid_base::{clone, sym};
+use orchid_extension::atom::{Atomic, TypAtom};
+use orchid_extension::atom_owned::{OwnedAtom, OwnedVariant};
+use orchid_extension::conv::{ToExpr, TryFromExpr};
+use orchid_extension::gen_expr::{atom, call, sym_ref};
+use orchid_extension::parser::{PSnippet, ParsCtx, ParsedLine, Parser};
+
+use crate::macros::let_line::{dealias_mac_v, parse_tokv};
+use crate::macros::mactree::{glossary_v, map_mactree_v};
+use crate::macros::recur_state::{RecurState, RulePath};
+use crate::macros::rule::matcher::{NamedMatcher, PriodMatcher};
+use crate::{Int, MacTok};
+
+#[derive(Default)]
+pub struct MacroLine;
+impl Parser for MacroLine {
+	const LINE_HEAD: &'static str = "macro";
+	async fn parse<'a>(
+		ctx: ParsCtx<'a>,
+		exported: bool,
+		comments: Vec<Comment>,
+		line: PSnippet<'a>,
+	) -> OrcRes<Vec<ParsedLine>> {
+		if exported {
+			return Err(mk_errv(
+				ctx.i().i("macros are always exported").await,
+				"The export keyword is forbidden here to avoid confusion\n\
+					because macros are exported by default",
+				[line.sr()],
+			));
+		}
+		let module = ctx.module();
+		let Parsed { output, tail } = try_pop_no_fluff(&ctx, line).await?;
+		let bad_first_item_err = || {
+			token_errv(&ctx, output, "Expected priority or block", |s| {
+				format!("Expected a priority number or a () block, found {s}")
+			})
+		};
+		let (prio, body) = match &output.tok {
+			Token::S(Paren::Round, body) => (None, body),
+			Token::Handle(expr) => match TypAtom::<Int>::try_from_expr(expr.clone()).await {
+				Err(e) => {
+					return Err(e + bad_first_item_err().await);
+				},
+				Ok(prio) => {
+					let Token::S(Paren::Round, block) = &output.tok else {
+						return Err(
+							token_errv(&ctx, output, "Expected () block", |s| {
+								format!("Expected a () block, found {s}")
+							})
+							.await,
+						);
+					};
+					(Some(prio), block)
+				},
+			},
+			_ => return Err(bad_first_item_err().await),
+		};
+		expect_end(&ctx, tail).await?;
+		let lines = line_items(&ctx, Snippet::new(output, body)).await;
+		let Some((kw_line, rule_lines)) = lines.split_first() else { return Ok(Vec::new()) };
+		let mut keywords = HashMap::new();
+		let Parsed { tail: kw_tail, .. } =
+			expect_tok(&ctx, kw_line.tail, ctx.i().i("keywords").await).await?;
+		for kw_tok in kw_tail.iter().filter(|kw| !kw.is_fluff()) {
+			match kw_tok.as_name() {
+				Some(kw) => {
+					keywords.insert(kw, kw_tok.sr());
+				},
+				None => ctx.rep().report(
+					token_errv(&ctx, kw_tok, "invalid macro keywords list", |tok| {
+						format!("The keywords list must be a sequence of names; received {tok}")
+					})
+					.await,
+				),
+			}
+		}
+		let Some(macro_name) = keywords.keys().next().cloned() else {
+			return Err(mk_errv(
+				ctx.i().i("macro with no keywords").await,
+				"Macros must define at least one macro of their own.",
+				[kw_line.tail.sr()],
+			));
+		};
+		let mut rules = Vec::new();
+		let mut lines = Vec::new();
+		for (idx, line) in rule_lines.iter().enumerate().map(|(n, v)| (n as u32, v)) {
+			let path = RulePath { module: module.clone(), main_kw: macro_name.clone(), rule: idx };
+			let sr = line.tail.sr();
+			let name = ctx.i().i(&path.name()).await;
+			let Parsed { tail, .. } = expect_tok(&ctx, line.tail, ctx.i().i("rule").await).await?;
+			let arrow_token = ctx.i().i("=>").await;
+			let Some((pattern, body)) = tail.split_once(|tok| tok.is_kw(arrow_token.clone())) else {
+				ctx.rep().report(mk_errv(
+					ctx.i().i("Missing => in rule").await,
+					"Rule lines are of the form `rule ...pattern => ...body`",
+					[line.tail.sr()],
+				));
+				continue;
+			};
+			let pattern = parse_tokv(pattern, &ctx).await;
+			let mut placeholders = Vec::new();
+			map_mactree_v(&pattern, &mut false, &mut |tok| {
+				if let MacTok::Ph(ph) = tok.tok() {
+					placeholders.push((ph.clone(), tok.pos()))
+				}
+				None
+			});
+			let mut body_mactree = parse_tokv(body, &ctx).await;
+			for (ph, ph_pos) in placeholders.iter().rev() {
+				let name = ctx.module().suffix([ph.name.clone()], ctx.i()).await;
+				body_mactree = vec![
+					MacTok::Lambda(MacTok::Name(name).at(ph_pos.clone()), body_mactree).at(ph_pos.clone()),
+				]
+			}
+			let body_sr = body.sr();
+			rules.push((name.clone(), placeholders, rules.len() as u32, sr.pos(), pattern));
+			lines.push(ParsedLine::cnst(&sr, &line.output, true, name, async move |ctx| {
+				let rep = Reporter::new();
+				let body = dealias_mac_v(body_mactree, &ctx, &rep).await;
+				let macro_input = MacTok::S(Paren::Round, body).at(body_sr.pos());
+				if let Some(e) = rep.errv() {
+					return Err(e);
+				}
+				Ok(call([
+					sym_ref(sym!(macro::resolve_recur; ctx.i()).await),
+					atom(RecurState::base(path)),
+					macro_input.to_expr().await,
+				]))
+			}))
+		}
+		let mac_cell = Rc::new(OnceCell::new());
+		let keywords = Rc::new(keywords);
+		let rules = Rc::new(RefCell::new(Some(rules)));
+		for (kw, sr) in &*keywords {
+			clone!(mac_cell, keywords, rules, module, prio);
+			lines.push(ParsedLine::cnst(&sr.clone(), &comments, true, kw.clone(), async move |cctx| {
+				let mac = mac_cell
+					.get_or_init(async {
+						let rep = Reporter::new();
+						let rules = rules.borrow_mut().take().expect("once cell initializer runs");
+						let rules = stream::from_iter(rules)
+							.then(|(body_name, placeholders, index, pos, pattern_macv)| {
+								let cctx = &cctx;
+								let rep = &rep;
+								let prio = &prio;
+								async move {
+									let pattern_abs = dealias_mac_v(pattern_macv, cctx, rep).await;
+									let glossary = glossary_v(&pattern_abs).collect();
+									let pattern_res = match prio {
+										None => NamedMatcher::new(&pattern_abs, cctx.i()).await.map(Matcher::Named),
+										Some(_) => PriodMatcher::new(&pattern_abs, cctx.i()).await.map(Matcher::Priod),
+									};
+									let placeholders = placeholders.into_iter().map(|(ph, _)| ph.name).collect_vec();
+									match pattern_res {
+										Ok(pattern) =>
+											Some(Rule { index, pos, body_name, pattern, glossary, placeholders }),
+										Err(e) => {
+											rep.report(e);
+											None
+										},
+									}
+								}
+							})
+							.flat_map(stream::from_iter)
+							.collect::<Vec<_>>()
+							.await;
+						let own_kws = keywords.keys().cloned().collect_vec();
+						Macro(Rc::new(MacroData { module, prio: prio.map(|i| i.0 as u64), rules, own_kws }))
+					})
+					.await;
+				atom(mac.clone())
+			}))
+		}
+		Ok(lines)
+	}
+}
+
+#[derive(Debug)]
+pub struct MacroData {
+	pub module: Sym,
+	pub prio: Option<u64>,
+	pub rules: Vec<Rule>,
+	pub own_kws: Vec<Tok<String>>,
+}
+
+#[derive(Clone, Debug)]
+pub struct Macro(pub Rc<MacroData>);
+#[derive(Debug)]
+pub struct Rule {
+	pub index: u32,
+	pub pos: Pos,
+	pub pattern: Matcher,
+	pub glossary: HashSet<Sym>,
+	pub placeholders: Vec<Tok<String>>,
+	pub body_name: Tok<String>,
+}
+#[derive(Debug)]
+pub enum Matcher {
+	Named(NamedMatcher),
+	Priod(PriodMatcher),
+}
+impl Atomic for Macro {
+	type Data = ();
+	type Variant = OwnedVariant;
+}
+impl OwnedAtom for Macro {
+	type Refs = Never;
+	async fn val(&self) -> Cow<'_, Self::Data> { Cow::Owned(()) }
+}

orchid-std/src/macros/macro_system.rs

@@ -2,7 +2,7 @@ use never::Never;
 use orchid_base::interner::Interner;
 use orchid_base::name::Sym;
 use orchid_base::reqnot::Receipt;
-use orchid_extension::atom::AtomDynfo;
+use orchid_extension::atom::{AtomDynfo, AtomicFeatures};
 use orchid_extension::entrypoint::ExtReq;
 use orchid_extension::lexer::LexerObj;
 use orchid_extension::parser::ParserObj;
@@ -10,14 +10,18 @@ use orchid_extension::system::{System, SystemCard};
 use orchid_extension::system_ctor::SystemCtor;
 use orchid_extension::tree::GenMember;
 
+use crate::macros::instantiate_tpl::InstantiateTplCall;
 use crate::macros::let_line::LetLine;
 use crate::macros::macro_lib::gen_macro_lib;
+use crate::macros::macro_line::MacroLine;
 use crate::macros::mactree_lexer::MacTreeLexer;
+use crate::macros::recur_state::RecurState;
+use crate::{MacTree, StdSystem};
 
 #[derive(Default)]
 pub struct MacroSystem;
 impl SystemCtor for MacroSystem {
-	type Deps = ();
+	type Deps = StdSystem;
 	type Instance = Self;
 	const NAME: &'static str = "macros";
 	const VERSION: f64 = 0.00_01;
@@ -26,12 +30,14 @@ impl SystemCtor for MacroSystem {
 impl SystemCard for MacroSystem {
 	type Ctor = Self;
 	type Req = Never;
-	fn atoms() -> impl IntoIterator<Item = Option<Box<dyn AtomDynfo>>> { [] }
+	fn atoms() -> impl IntoIterator<Item = Option<Box<dyn AtomDynfo>>> {
+		[Some(InstantiateTplCall::dynfo()), Some(MacTree::dynfo()), Some(RecurState::dynfo())]
+	}
 }
 impl System for MacroSystem {
 	async fn request(_: ExtReq<'_>, req: Self::Req) -> Receipt<'_> { match req {} }
 	async fn prelude(_: &Interner) -> Vec<Sym> { vec![] }
 	fn lexers() -> Vec<LexerObj> { vec![&MacTreeLexer] }
-	fn parsers() -> Vec<ParserObj> { vec![&LetLine] }
+	fn parsers() -> Vec<ParserObj> { vec![&LetLine, &MacroLine] }
 	fn env() -> Vec<GenMember> { gen_macro_lib() }
 }

orchid-std/src/macros/mactree.rs

@@ -4,10 +4,11 @@ use std::rc::Rc;
 
 use futures::FutureExt;
 use futures::future::join_all;
+use hashbrown::HashSet;
 use itertools::Itertools;
 use orchid_api::Paren;
-use orchid_base::error::OrcErrv;
+use orchid_base::error::{OrcErrv, Reporter, mk_errv};
-use orchid_base::format::{FmtCtx, FmtUnit, Format, Variants};
+use orchid_base::format::{FmtCtx, FmtUnit, Format, Variants, fmt};
 use orchid_base::interner::Tok;
 use orchid_base::location::Pos;
 use orchid_base::name::Sym;
@@ -15,16 +16,71 @@ use orchid_base::tl_cache;
 use orchid_base::tree::indent;
 use orchid_extension::atom::Atomic;
 use orchid_extension::atom_owned::{OwnedAtom, OwnedVariant};
+use orchid_extension::conv::ToExpr;
 use orchid_extension::expr::Expr;
+use orchid_extension::gen_expr::{GExpr, arg, bot, call, lambda, sym_ref};
+use orchid_extension::system::SysCtx;
+use substack::Substack;
+
+#[derive(Clone)]
+pub struct LowerCtx<'a> {
+	pub sys: SysCtx,
+	pub rep: &'a Reporter,
+}
 
 #[derive(Debug, Clone)]
 pub struct MacTree {
 	pub pos: Pos,
 	pub tok: Rc<MacTok>,
+	pub glossary: Rc<HashSet<Sym>>,
 }
 impl MacTree {
-	pub fn tok(&self) -> &MacTok { &*self.tok }
+	pub fn tok(&self) -> &MacTok { &self.tok }
 	pub fn pos(&self) -> Pos { self.pos.clone() }
+	pub fn glossary(&self) -> &HashSet<Sym> { &self.glossary }
+	pub async fn lower(&self, ctx: LowerCtx<'_>, args: Substack<'_, Sym>) -> GExpr {
+		let expr = match self.tok() {
+			MacTok::Bottom(e) => bot(e.clone()),
+			MacTok::Lambda(arg, body) => {
+				let MacTok::Name(name) = &*arg.tok else {
+					let err = mk_errv(
+						ctx.sys.i().i("Syntax error after macros").await,
+						"This token ends up as a binding, consider replacing it with a name",
+						[arg.pos()],
+					);
+					ctx.rep.report(err.clone());
+					return bot(err);
+				};
+				lambda(args.len() as u64, lower_v(body, ctx, args.push(name.clone())).await)
+			},
+			MacTok::Name(name) => match args.iter().enumerate().find(|(_, n)| *n == name) {
+				None => sym_ref(name.clone()),
+				Some((i, _)) => arg((args.len() - i) as u64),
+			},
+			MacTok::Ph(ph) => {
+				let err = mk_errv(
+					ctx.sys.i().i("Placeholder in value").await,
+					format!("Placeholder {ph} is only supported in macro patterns"),
+					[self.pos()],
+				);
+				ctx.rep.report(err.clone());
+				return bot(err);
+			},
+			MacTok::S(Paren::Round, body) => call(lower_v(body, ctx, args).await),
+			MacTok::S(..) => {
+				let err = mk_errv(
+					ctx.sys.i().i("[] or {} after macros").await,
+					format!("{} didn't match any macro", fmt(self, ctx.sys.i()).await),
+					[self.pos()],
+				);
+				ctx.rep.report(err.clone());
+				return bot(err);
+			},
+			MacTok::Slot => panic!("Uninstantiated template should never be exposed"),
+			MacTok::Value(v) => v.clone().to_expr().await,
+		};
+		expr.at(self.pos())
+	}
 }
 impl Atomic for MacTree {
 	type Data = ();
@@ -34,11 +90,20 @@ impl OwnedAtom for MacTree {
 	type Refs = ();
 
 	async fn val(&self) -> Cow<'_, Self::Data> { Cow::Owned(()) }
+	async fn print_atom<'a>(&'a self, c: &'a (impl FmtCtx + ?Sized + 'a)) -> FmtUnit {
+		self.tok.print(c).await
+	}
+}
+impl Format for MacTree {
 	async fn print<'a>(&'a self, c: &'a (impl FmtCtx + ?Sized + 'a)) -> FmtUnit {
 		self.tok.print(c).await
 	}
 }
 
+pub async fn lower_v(v: &[MacTree], ctx: LowerCtx<'_>, args: Substack<'_, Sym>) -> Vec<GExpr> {
+	join_all(v.iter().map(|t| t.lower(ctx.clone(), args.clone())).collect::<Vec<_>>()).await
+}
+
 #[derive(Debug, Clone)]
 pub enum MacTok {
 	S(Paren, Vec<MacTree>),
@@ -53,8 +118,17 @@ pub enum MacTok {
 	Bottom(OrcErrv),
 }
 impl MacTok {
+	pub fn build_glossary(&self) -> HashSet<Sym> {
+		match self {
+			MacTok::Bottom(_) | MacTok::Ph(_) | MacTok::Slot | MacTok::Value(_) => HashSet::new(),
+			MacTok::Name(sym) => HashSet::from([sym.clone()]),
+			MacTok::S(_, body) => body.iter().flat_map(|mt| &*mt.glossary).cloned().collect(),
+			MacTok::Lambda(arg, body) =>
+				body.iter().chain([arg]).flat_map(|mt| &*mt.glossary).cloned().collect(),
+		}
+	}
 	pub fn at(self, pos: impl Into<Pos>) -> MacTree {
-		MacTree { pos: pos.into(), tok: Rc::new(self) }
+		MacTree { pos: pos.into(), glossary: Rc::new(self.build_glossary()), tok: Rc::new(self) }
 	}
 }
 impl Format for MacTok {
@@ -77,7 +151,7 @@ impl Format for MacTok {
 				},
 				[mtreev_fmt(body, c).await],
 			),
-			Self::Slot => "SLOT".into(),
+			Self::Slot => "$SLOT".into(),
 			Self::Bottom(err) if err.len() == 1 => format!("Bottom({}) ", err.one().unwrap()).into(),
 			Self::Bottom(err) => format!("Botttom(\n{}) ", indent(&err.to_string())).into(),
 		}
@@ -129,12 +203,12 @@ pub fn map_mactree<F: FnMut(MacTree) -> Option<MacTree>>(
 				ro(changed, |changed| map_mactree(arg, changed, map)),
 				map_mactree_v(body, changed, map),
 			),
-			MacTok::Name(_) | MacTok::Value(_) | MacTok::Slot | MacTok::Ph(_) | MacTok::Bottom(_) =>
+			MacTok::Name(_) | MacTok::Value(_) => return src.clone(),
-				return src.clone(),
+			MacTok::Slot | MacTok::Ph(_) | MacTok::Bottom(_) => return src.clone(),
 			MacTok::S(p, body) => MacTok::S(*p, map_mactree_v(body, changed, map)),
 		},
 	};
-	if *changed { MacTree { pos: src.pos.clone(), tok: Rc::new(tok) } } else { src.clone() }
+	if *changed { tok.at(src.pos()) } else { src.clone() }
 }
 pub fn map_mactree_v<F: FnMut(MacTree) -> Option<MacTree>>(
 	src: &[MacTree],
@@ -152,3 +226,7 @@ fn ro<T>(flag: &mut bool, cb: impl FnOnce(&mut bool) -> T) -> T {
 	*flag |= new_flag;
 	val
 }
+
+pub fn glossary_v(src: &[MacTree]) -> impl Iterator<Item = Sym> {
+	src.iter().flat_map(|mt| mt.glossary()).cloned()
+}

orchid-std/src/macros/mactree_lexer.rs

@@ -1,12 +1,16 @@
 use std::ops::RangeInclusive;
-use std::rc::Rc;
 
 use futures::FutureExt;
 use orchid_base::error::{OrcRes, mk_errv};
+use orchid_base::parse::ParseCtx;
+use orchid_base::sym;
 use orchid_base::tokens::PARENS;
+use orchid_base::tree::Paren;
 use orchid_extension::lexer::{LexContext, Lexer, err_not_applicable};
-use orchid_extension::tree::{GenTok, GenTokTree, x_tok};
+use orchid_extension::parser::p_tree2gen;
+use orchid_extension::tree::{GenTok, GenTokTree, ref_tok, x_tok};
 
+use crate::macros::let_line::parse_tok;
 use crate::macros::mactree::{MacTok, MacTree};
 
 #[derive(Default)]
@@ -15,24 +19,41 @@ impl Lexer for MacTreeLexer {
 	const CHAR_FILTER: &'static [RangeInclusive<char>] = &['\''..='\''];
 	async fn lex<'a>(tail: &'a str, ctx: &'a LexContext<'a>) -> OrcRes<(&'a str, GenTokTree)> {
 		let Some(tail2) = tail.strip_prefix('\'') else {
-			return Err(err_not_applicable(ctx.i()).await.into());
+			return Err(err_not_applicable(ctx.i()).await);
 		};
 		let tail3 = tail2.trim_start();
-		return match mac_tree(tail3, ctx).await {
+		let mut args = Vec::new();
-			Ok((tail4, mactree)) => Ok((tail4, x_tok(mactree).at(ctx.pos_tt(tail, tail4)))),
+		return match mac_tree(tail3, &mut args, ctx).await {
+			Ok((tail4, mactree)) => {
+				let range = ctx.pos_tt(tail, tail4);
+				let tok = match &args[..] {
+					[] => x_tok(mactree).await,
+					_ => {
+						let call = ([
+							ref_tok(sym!(macros::instantiate_tpl; ctx.i()).await).await.at(range.clone()),
+							x_tok(mactree).await.at(range.clone()),
+						]
+						.into_iter())
+						.chain(args.into_iter());
+						GenTok::S(Paren::Round, call.collect())
+					},
+				};
+				Ok((tail4, tok.at(range)))
+			},
 			Err(e) => Ok((tail2, GenTok::Bottom(e).at(ctx.pos_lt(1, tail2)))),
 		};
-		async fn mac_tree<'a>(tail: &'a str, ctx: &'a LexContext<'a>) -> OrcRes<(&'a str, MacTree)> {
+		async fn mac_tree<'a>(
+			tail: &'a str,
+			args: &mut Vec<GenTokTree>,
+			ctx: &'a LexContext<'a>,
+		) -> OrcRes<(&'a str, MacTree)> {
 			for (lp, rp, paren) in PARENS {
 				let Some(mut body_tail) = tail.strip_prefix(*lp) else { continue };
 				let mut items = Vec::new();
 				return loop {
-					let tail2 = body_tail.trim();
+					let tail2 = body_tail.trim_start();
 					if let Some(tail3) = tail2.strip_prefix(*rp) {
-						break Ok((tail3, MacTree {
+						break Ok((tail3, MacTok::S(*paren, items).at(ctx.pos_tt(tail, tail3).pos())));
-							pos: ctx.pos_tt(tail, tail3).pos(),
-							tok: Rc::new(MacTok::S(*paren, items)),
-						}));
 					} else if tail2.is_empty() {
 						return Err(mk_errv(
 							ctx.i().i("Unclosed block").await,
@@ -40,22 +61,36 @@ impl Lexer for MacTreeLexer {
 							[ctx.pos_lt(1, tail)],
 						));
 					}
-					let (new_tail, new_item) = mac_tree(tail2, ctx).boxed_local().await?;
+					let (new_tail, new_item) = mac_tree(tail2, args, ctx).boxed_local().await?;
 					body_tail = new_tail;
 					items.push(new_item);
 				};
 			}
-			const INTERPOL: &[&str] = &["$", "..$"];
+			if let Some(tail2) = tail.strip_prefix("$") {
-			for pref in INTERPOL {
+				let (tail3, sub) = ctx.recurse(tail2).await?;
-				let Some(code) = tail.strip_prefix(pref) else { continue };
+				let sr = ctx.pos_tt(tail, tail3);
-				todo!("Register parameter, and push this onto the argument stack held in the atom")
+				args.push(p_tree2gen(sub));
+				return Ok((tail3, MacTok::Slot.at(sr.pos())));
+			}
+			if let Some(tail2) = tail.strip_prefix("\\") {
+				let tail2 = tail2.trim_start();
+				let (mut tail3, param) = mac_tree(tail2, args, ctx).boxed_local().await?;
+				let mut body = Vec::new();
+				loop {
+					let tail4 = tail3.trim_start();
+					if tail4.is_empty() || tail4.starts_with(|c| ")]}".contains(c)) {
+						break;
+					};
+					let (tail5, body_tok) = mac_tree(tail4, args, ctx).boxed_local().await?;
+					body.push(body_tok);
+					tail3 = tail5;
+				}
+				Ok((tail3, MacTok::Lambda(param, body).at(ctx.pos_tt(tail, tail3).pos())))
+			} else {
+				let (tail2, sub) = ctx.recurse(tail).await?;
+				let parsed = parse_tok(&sub, ctx).await.expect("Unexpected invalid token");
+				Ok((tail2, parsed))
 			}
-			todo!("recursive lexer call");
-			return Err(mk_errv(
-				ctx.i().i("Expected token after '").await,
-				format!("Expected a token after ', found {tail:?}"),
-				[ctx.pos_lt(1, tail)],
-			));
 		}
 	}
 }

orchid-std/src/macros/mod.rs

@@ -1,9 +1,12 @@
 mod instantiate_tpl;
 mod let_line;
 mod macro_lib;
-mod macro_system;
+mod macro_line;
+pub mod macro_system;
 pub mod mactree;
 mod mactree_lexer;
+pub mod recur_state;
+mod resolve;
 mod rule;
 
 use mactree::{MacTok, MacTree};

orchid-std/src/macros/recur_state.rs

@@ -0,0 +1,59 @@
+use std::borrow::Cow;
+use std::fmt;
+use std::rc::Rc;
+
+use never::Never;
+use orchid_base::interner::Tok;
+use orchid_base::name::Sym;
+use orchid_extension::atom::Atomic;
+use orchid_extension::atom_owned::{OwnedAtom, OwnedVariant};
+
+#[derive(Clone, PartialEq, Eq, Hash)]
+pub struct RulePath {
+	pub module: Sym,
+	pub main_kw: Tok<String>,
+	pub rule: u32,
+}
+impl RulePath {
+	pub fn name(&self) -> String { format!("rule::{}::{}", self.main_kw, self.rule) }
+}
+impl fmt::Display for RulePath {
+	fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+		write!(f, "Rule {}::({})::{}", self.module, self.main_kw, self.rule)
+	}
+}
+
+#[derive(Clone)]
+pub enum RecurState {
+	Bottom,
+	Recursive { path: RulePath, prev: Rc<RecurState> },
+}
+impl RecurState {
+	pub fn base(path: RulePath) -> Self {
+		RecurState::Recursive { path, prev: Rc::new(RecurState::Bottom) }
+	}
+	pub fn push(&self, new: RulePath) -> Option<Self> {
+		let mut cur = self;
+		while let Self::Recursive { path, prev } = cur {
+			if &new == path {
+				return None;
+			}
+			cur = prev;
+		}
+		Some(Self::Recursive { path: new, prev: Rc::new(self.clone()) })
+	}
+}
+impl Atomic for RecurState {
+	type Data = Option<()>;
+	type Variant = OwnedVariant;
+}
+impl OwnedAtom for RecurState {
+	type Refs = Never;
+
+	async fn val(&self) -> Cow<'_, Self::Data> {
+		Cow::Owned(match self {
+			Self::Bottom => None,
+			Self::Recursive { .. } => Some(()),
+		})
+	}
+}

orchid-std/src/macros/resolve.rs

@@ -0,0 +1,107 @@
+use futures::FutureExt;
+use hashbrown::HashMap;
+use itertools::Itertools;
+use orchid_base::error::mk_errv;
+use orchid_base::location::Pos;
+use orchid_base::name::Sym;
+use orchid_base::sym;
+use orchid_base::tree::Paren;
+use orchid_extension::conv::ToExpr;
+use orchid_extension::coroutine_exec::ExecHandle;
+use orchid_extension::gen_expr::{GExpr, bot, call, sym_ref};
+use orchid_extension::system::SysCtx;
+
+use crate::macros::macro_line::{Macro, Rule};
+use crate::macros::recur_state::{RecurState, RulePath};
+use crate::macros::rule::matcher::{NamedMatcher, PriodMatcher};
+use crate::macros::rule::state::{MatchState, StateEntry};
+use crate::{MacTok, MacTree};
+
+pub struct ResolveCtx<'a> {
+	pub ctx: SysCtx,
+	pub recur: RecurState,
+	pub h: ExecHandle<'a>,
+	pub named: HashMap<Sym, Vec<(&'a NamedMatcher, &'a Macro, &'a Rule)>>,
+	pub priod: Vec<(&'a PriodMatcher, &'a Macro, &'a Rule)>,
+}
+
+pub async fn resolve(ctx: &mut ResolveCtx<'_>, value: &MacTree) -> Option<MacTree> {
+	match value.tok() {
+		MacTok::Ph(_) | MacTok::Slot => panic!("Forbidden element in value mactree"),
+		MacTok::Bottom(_) | MacTok::Value(_) | MacTok::Name(_) => None,
+		MacTok::Lambda(arg, body) =>
+			Some(MacTok::Lambda(arg.clone(), resolve_seq(ctx, body).await?).at(value.pos())),
+		MacTok::S(ptyp, body) => Some(MacTok::S(*ptyp, resolve_seq(ctx, body).await?).at(value.pos())),
+	}
+}
+
+pub async fn resolve_seq(ctx: &mut ResolveCtx<'_>, val: &[MacTree]) -> Option<Vec<MacTree>> {
+	let mut any_changed = false;
+	let mut i = 0;
+	let mut val = val.to_vec();
+	while i < val.len() {
+		let MacTok::Name(key) = val[i].tok() else { continue };
+		let Some(options) = ctx.named.get(key) else { continue };
+		let matches = (options.iter())
+			.filter_map(|r| Some((r.1, r.2, r.0.apply(&val[i..], |_| false)?)))
+			.collect_vec();
+		match matches.len() {
+			0 => i += 1,
+			1 => {
+				any_changed = true;
+				let (mac, rule, (state, tail)) = matches.into_iter().exactly_one().unwrap();
+				let end = val.len() - tail.len();
+				let new_i = val.len() - tail.len();
+				let body_call = mk_body_call(mac, rule, &state, &ctx.ctx, ctx.recur.clone()).await;
+				std::mem::drop(state);
+				val.splice(i..end, [MacTok::Value(ctx.h.register(body_call).await).at(Pos::None)]);
+				i = new_i;
+			},
+			2.. => todo!("Named macros conflict!"),
+		}
+	}
+	for (matcher, mac, rule) in &ctx.priod {
+		let Some(state) = matcher.apply(&val, |_| false) else { continue };
+		return Some(vec![
+			MacTok::Value(
+				ctx.h.register(mk_body_call(mac, rule, &state, &ctx.ctx, ctx.recur.clone()).await).await,
+			)
+			.at(Pos::None),
+		]);
+	}
+	for expr in val.iter_mut() {
+		if let Some(new) = resolve(ctx, expr).boxed_local().await {
+			*expr = new;
+			any_changed = true;
+		}
+	}
+	if any_changed { Some(val) } else { None }
+}
+
+async fn mk_body_call(
+	mac: &Macro,
+	rule: &Rule,
+	state: &MatchState<'_>,
+	ctx: &SysCtx,
+	recur: RecurState,
+) -> GExpr {
+	let rule_path =
+		RulePath { module: mac.0.module.clone(), main_kw: mac.0.own_kws[0].clone(), rule: rule.index };
+	let Some(new_recur) = recur.push(rule_path.clone()) else {
+		return bot(mk_errv(
+			ctx.i().i("Circular macro dependency").await,
+			format!("The definition of {rule_path} is circular"),
+			[rule.pos.clone()],
+		));
+	};
+	let mut call_args = vec![sym_ref(mac.0.module.suffix([rule.body_name.clone()], ctx.i()).await)];
+	for name in rule.placeholders.iter() {
+		call_args.push(match state.get(name).expect("Missing state entry for placeholder") {
+			StateEntry::Scalar(scal) => (**scal).clone().to_expr().await,
+			StateEntry::Vec(vec) => MacTok::S(Paren::Round, vec.to_vec()).at(Pos::None).to_expr().await,
+		});
+	}
+	call_args
+		.push(call([sym_ref(sym!(macros::resolve_recur; ctx.i()).await), new_recur.to_expr().await]));
+	call(call_args)
+}

orchid-std/src/macros/rule/build.rs

@@ -16,7 +16,7 @@ pub type MaxVecSplit<'a> = (&'a [MacTree], (Tok<String>, u8, bool), &'a [MacTree
 /// Derive the details of the central vectorial and the two sides from a
 /// slice of Expr's
 #[must_use]
-fn split_at_max_vec(pattern: &[MacTree]) -> Option<MaxVecSplit> {
+fn split_at_max_vec(pattern: &'_ [MacTree]) -> Option<MaxVecSplit<'_>> {
 	let rngidx = pattern
 		.iter()
 		.position_max_by_key(|expr| vec_attrs(expr).map(|attrs| attrs.1 as i64).unwrap_or(-1))?;
@@ -31,7 +31,6 @@ fn scal_cnt<'a>(iter: impl Iterator<Item = &'a MacTree>) -> usize {
 	iter.take_while(|expr| vec_attrs(expr).is_none()).count()
 }
 
-#[must_use]
 pub async fn mk_any(pattern: &[MacTree], i: &Interner) -> OrcRes<AnyMatcher> {
 	let left_split = scal_cnt(pattern.iter());
 	if pattern.len() <= left_split {
@@ -49,13 +48,11 @@ pub async fn mk_any(pattern: &[MacTree], i: &Interner) -> OrcRes<AnyMatcher> {
 }
 
 /// Pattern MUST NOT contain vectorial placeholders
-#[must_use]
 async fn mk_scalv(pattern: &[MacTree], i: &Interner) -> OrcRes<Vec<ScalMatcher>> {
 	join_all(pattern.iter().map(|pat| mk_scalar(pat, i))).await.into_iter().collect()
 }
 
 /// Pattern MUST start and end with a vectorial placeholder
-#[must_use]
 pub async fn mk_vec(pattern: &[MacTree], i: &Interner) -> OrcRes<VecMatcher> {
 	debug_assert!(!pattern.is_empty(), "pattern cannot be empty");
 	debug_assert!(pattern.first().map(vec_attrs).is_some(), "pattern must start with a vectorial");
@@ -116,7 +113,6 @@ pub async fn mk_vec(pattern: &[MacTree], i: &Interner) -> OrcRes<VecMatcher> {
 }
 
 /// Pattern MUST NOT be a vectorial placeholder
-#[must_use]
 async fn mk_scalar(pattern: &MacTree, i: &Interner) -> OrcRes<ScalMatcher> {
 	Ok(match &*pattern.tok {
 		MacTok::Name(n) => ScalMatcher::Name(n.clone()),
@@ -140,8 +136,6 @@ async fn mk_scalar(pattern: &MacTree, i: &Interner) -> OrcRes<ScalMatcher> {
 
 #[cfg(test)]
 mod test {
-	use std::rc::Rc;
-
 	use orchid_base::interner::Interner;
 	use orchid_base::location::SrcRange;
 	use orchid_base::sym;
@@ -149,15 +143,14 @@ mod test {
 	use test_executors::spin_on;
 
 	use super::mk_any;
+	use crate::macros::MacTok;
 	use crate::macros::mactree::{Ph, PhKind};
-	use crate::macros::{MacTok, MacTree};
 
 	#[test]
 	fn test_scan() {
 		spin_on(async {
 			let i = Interner::new_master();
-			let ex =
+			let ex = |tok: MacTok| async { tok.at(SrcRange::mock(&i).await.pos()) };
-				|tok: MacTok| async { MacTree { tok: Rc::new(tok), pos: SrcRange::mock(&i).await.pos() } };
 			let pattern = vec![
 				ex(MacTok::Ph(Ph {
 					kind: PhKind::Vector { priority: 0, at_least_one: false },

orchid-std/src/macros/rule/matcher.rs

@@ -1,9 +1,8 @@
 use std::fmt;
-use std::rc::Rc;
 
 use itertools::Itertools;
 use orchid_base::error::OrcRes;
-use orchid_base::interner::Interner;
+use orchid_base::interner::{Interner, Tok};
 use orchid_base::location::Pos;
 use orchid_base::name::Sym;
 
@@ -16,46 +15,47 @@ use super::vec_match::vec_match;
 use crate::macros::mactree::{Ph, PhKind};
 use crate::macros::{MacTok, MacTree};
 
-pub fn first_is_vec(pattern: &[MacTree]) -> bool { vec_attrs(pattern.first().unwrap()).is_some() }
+pub struct NamedMatcher {
-pub fn last_is_vec(pattern: &[MacTree]) -> bool { vec_attrs(pattern.last().unwrap()).is_some() }
+	inner: AnyMatcher,
-
+	head: Sym,
-pub struct NamedMatcher(AnyMatcher);
+	after_tok: Tok<String>,
+}
 impl NamedMatcher {
 	pub async fn new(pattern: &[MacTree], i: &Interner) -> OrcRes<Self> {
-		assert!(
+		let head = match pattern.first().map(|tree| tree.tok()) {
-			matches!(pattern.first().map(|tree| &*tree.tok), Some(MacTok::Name(_))),
+			Some(MacTok::Name(name)) => name.clone(),
-			"Named matchers must begin with a name"
+			_ => panic!("Named matchers must begin with a name"),
-		);
+		};
-
+		let after_tok = i.i("::after").await;
-		Ok(Self(match last_is_vec(pattern) {
+		let inner = match pattern.last().and_then(vec_attrs).is_some() {
-			true => mk_any(pattern, i).await,
+			true => mk_any(pattern, i).await?,
 			false => {
 				let kind = PhKind::Vector { priority: 0, at_least_one: false };
-				let tok = MacTok::Ph(Ph { name: i.i("::after").await, kind });
+				let suffix = [MacTok::Ph(Ph { name: after_tok.clone(), kind }).at(Pos::None)];
-				let suffix = [MacTree { pos: Pos::None, tok: Rc::new(tok) }];
+				mk_any(&pattern.iter().cloned().chain(suffix).collect_vec(), i).await?
-				mk_any(&pattern.iter().chain(&suffix).cloned().collect_vec(), i).await
 			},
-		}?))
+		};
+		Ok(Self { after_tok, inner, head })
 	}
+	pub fn head(&self) -> Sym { self.head.clone() }
 	/// Also returns the tail, if any, which should be matched further
 	/// Note that due to how priod works below, the main usable information from
 	/// the tail is its length
-	pub async fn apply<'a>(
+	pub fn apply<'a>(
 		&self,
 		seq: &'a [MacTree],
-		i: &Interner,
 		save_loc: impl Fn(Sym) -> bool,
 	) -> Option<(MatchState<'a>, &'a [MacTree])> {
-		let mut state = any_match(&self.0, seq, &save_loc)?;
+		let mut state = any_match(&self.inner, seq, &save_loc)?;
-		match state.remove(i.i("::after").await) {
+		match state.remove(self.after_tok.clone()) {
-			Some(StateEntry::Scalar(_)) => panic!("::after can never be a scalar entry!"),
+			Some(StateEntry::Scalar(_)) => panic!("{} can never be a scalar entry!", self.after_tok),
 			Some(StateEntry::Vec(v)) => Some((state, v)),
 			None => Some((state, &[][..])),
 		}
 	}
 }
 impl fmt::Display for NamedMatcher {
-	fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { self.0.fmt(f) }
+	fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { self.inner.fmt(f) }
 }
 impl fmt::Debug for NamedMatcher {
 	fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { write!(f, "NamedMatcher({self})") }

orchid-std/src/macros/rule/state.rs

@@ -54,6 +54,7 @@ impl<'a> MatchState<'a> {
 	pub fn from_name(name: Sym, location: Pos) -> Self {
 		Self { name_posv: HashMap::from([(name, vec![location])]), placeholders: HashMap::new() }
 	}
+	pub fn get(&self, key: &Tok<String>) -> Option<&StateEntry<'a>> { self.placeholders.get(key) }
 	pub fn remove(&mut self, name: Tok<String>) -> Option<StateEntry<'a>> {
 		self.placeholders.remove(&name)
 	}

orchid-std/src/main.rs

@@ -1,6 +1,8 @@
 use orchid_extension::entrypoint::ExtensionData;
 use orchid_extension::tokio::tokio_main;
-use orchid_std::StdSystem;
+use orchid_std::{MacroSystem, StdSystem};
 
 #[tokio::main(flavor = "current_thread")]
-pub async fn main() { tokio_main(ExtensionData::new("orchid-std::main", &[&StdSystem])).await }
+pub async fn main() {
+	tokio_main(ExtensionData::new("orchid-std::main", &[&StdSystem, &MacroSystem])).await
+}

orchid-std/src/std/number/num_lexer.rs

@@ -1,7 +1,7 @@
 use std::ops::RangeInclusive;
 
 use orchid_base::error::OrcRes;
-use orchid_base::number::{num_to_err, parse_num};
+use orchid_base::number::{num_to_errv, parse_num};
 use orchid_extension::atom::ToAtom;
 use orchid_extension::lexer::{LexContext, Lexer};
 use orchid_extension::tree::{GenTokTree, x_tok};
@@ -17,8 +17,8 @@ impl Lexer for NumLexer {
 		let (chars, tail) = all.split_at(ends_at.unwrap_or(all.len()));
 		let fac = match parse_num(chars) {
 			Ok(numeric) => Num(numeric).to_atom_factory(),
-			Err(e) => return Err(num_to_err(e, ctx.pos(all), &ctx.src, ctx.ctx.i()).await.into()),
+			Err(e) => return Err(num_to_errv(e, ctx.pos(all), ctx.src(), ctx.ctx.i()).await),
 		};
-		Ok((tail, x_tok(fac).at(ctx.pos_lt(chars.len(), tail))))
+		Ok((tail, x_tok(fac).await.at(ctx.pos_lt(chars.len(), tail))))
 	}
 }

orchid-std/src/std/string/str_atom.rs

@@ -49,7 +49,7 @@ impl OwnedAtom for StrAtom {
 	async fn serialize(&self, _: SysCtx, sink: Pin<&mut (impl Write + ?Sized)>) -> Self::Refs {
 		self.deref().encode(sink).await
 	}
-	async fn print<'a>(&'a self, _: &'a (impl FmtCtx + ?Sized + 'a)) -> FmtUnit {
+	async fn print_atom<'a>(&'a self, _: &'a (impl FmtCtx + ?Sized + 'a)) -> FmtUnit {
 		format!("{:?}", &*self.0).into()
 	}
 	async fn deserialize(mut ctx: impl DeserializeCtx, _: Self::Refs) -> Self {
@@ -69,7 +69,7 @@ impl From<Tok<String>> for IntStrAtom {
 impl OwnedAtom for IntStrAtom {
 	type Refs = ();
 	async fn val(&self) -> Cow<'_, Self::Data> { Cow::Owned(self.0.to_api()) }
-	async fn print<'a>(&'a self, _: &'a (impl FmtCtx + ?Sized + 'a)) -> FmtUnit {
+	async fn print_atom<'a>(&'a self, _: &'a (impl FmtCtx + ?Sized + 'a)) -> FmtUnit {
 		format!("{:?}i", *self.0).into()
 	}
 	async fn serialize(&self, _: SysCtx, write: Pin<&mut (impl Write + ?Sized)>) {
@@ -108,7 +108,7 @@ impl TryFromExpr for OrcString {
 		}
 		let ctx = expr.ctx();
 		match TypAtom::<IntStrAtom>::try_from_expr(expr).await {
-			Ok(t) => Ok(OrcString { ctx: t.data.ctx(), kind: OrcStringKind::Int(t) }),
+			Ok(t) => Ok(OrcString { ctx: t.untyped.ctx().clone(), kind: OrcStringKind::Int(t) }),
 			Err(e) => Err(mk_errv(ctx.i().i("A string was expected").await, "", e.pos_iter())),
 		}
 	}

orchid-std/src/std/string/str_lexer.rs

@@ -1,11 +1,13 @@
 use itertools::Itertools;
-use orchid_base::error::{OrcErr, OrcRes, mk_err, mk_errv};
+use orchid_base::error::{OrcErr, OrcErrv, OrcRes, mk_errv};
 use orchid_base::interner::Interner;
 use orchid_base::location::SrcRange;
 use orchid_base::name::Sym;
+use orchid_base::parse::ParseCtx;
 use orchid_base::sym;
 use orchid_base::tree::wrap_tokv;
 use orchid_extension::lexer::{LexContext, Lexer, err_not_applicable};
+use orchid_extension::parser::p_tree2gen;
 use orchid_extension::tree::{GenTokTree, ref_tok, x_tok};
 
 use super::str_atom::IntStrAtom;
@@ -32,16 +34,16 @@ struct StringError {
 
 impl StringError {
 	/// Convert into project error for reporting
-	pub async fn into_proj(self, path: &Sym, pos: u32, i: &Interner) -> OrcErr {
+	pub async fn into_proj(self, path: &Sym, pos: u32, i: &Interner) -> OrcErrv {
 		let start = pos + self.pos;
-		mk_err(
+		mk_errv(
 			i.i("Failed to parse string").await,
 			match self.kind {
 				StringErrorKind::NotHex => "Expected a hex digit",
 				StringErrorKind::BadCodePoint => "The specified number is not a Unicode code point",
 				StringErrorKind::BadEscSeq => "Unrecognized escape sequence",
 			},
-			[SrcRange::new(start..start + 1, path).pos().into()],
+			[SrcRange::new(start..start + 1, path).pos()],
 		)
 	}
 }
@@ -97,7 +99,7 @@ impl Lexer for StringLexer {
 	const CHAR_FILTER: &'static [std::ops::RangeInclusive<char>] = &['"'..='"', '`'..='`'];
 	async fn lex<'a>(all: &'a str, ctx: &'a LexContext<'a>) -> OrcRes<(&'a str, GenTokTree)> {
 		let Some(mut tail) = all.strip_prefix('"') else {
-			return Err(err_not_applicable(ctx.ctx.i()).await.into());
+			return Err(err_not_applicable(ctx.ctx.i()).await);
 		};
 		let mut ret = None;
 		let mut cur = String::new();
@@ -110,15 +112,17 @@ impl Lexer for StringLexer {
 		) -> GenTokTree {
 			let str_val_res = parse_string(&str.split_off(0));
 			if let Err(e) = &str_val_res {
-				err.push(e.clone().into_proj(&ctx.src, ctx.pos(tail) - str.len() as u32, ctx.i()).await);
+				err.extend(e.clone().into_proj(ctx.src(), ctx.pos(tail) - str.len() as u32, ctx.i()).await);
 			}
 			let str_val = str_val_res.unwrap_or_default();
-			x_tok(IntStrAtom::from(ctx.i().i(&*str_val).await)).at(ctx.pos_lt(str.len() as u32, tail))
+			x_tok(IntStrAtom::from(ctx.i().i(&*str_val).await))
-				as GenTokTree
+				.await
+				.at(ctx.pos_lt(str.len() as u32, tail)) as GenTokTree
 		}
 		let add_frag = |prev: Option<GenTokTree>, new: GenTokTree| async {
 			let Some(prev) = prev else { return new };
 			let concat_fn = ref_tok(sym!(std::string::concat; ctx.i()).await)
+				.await
 				.at(SrcRange::zw(prev.sr.path(), prev.sr.start()));
 			wrap_tokv([concat_fn, prev, new])
 		};
@@ -129,7 +133,7 @@ impl Lexer for StringLexer {
 				ret = Some(add_frag(ret, str_to_gen(&mut cur, tail, &mut errors, ctx).await).await);
 				let (new_tail, tree) = ctx.recurse(rest).await?;
 				tail = new_tail;
-				ret = Some(add_frag(ret, tree).await);
+				ret = Some(add_frag(ret, p_tree2gen(tree)).await);
 			} else if tail.starts_with('\\') {
 				// parse_string will deal with it, we just have to skip the next char
 				tail = &tail[2..];
@@ -143,7 +147,7 @@ impl Lexer for StringLexer {
 					return Err(mk_errv(
 						ctx.i().i("No string end").await,
 						"String never terminated with \"",
-						[SrcRange::new(range.clone(), &ctx.src)],
+						[SrcRange::new(range.clone(), ctx.src())],
 					));
 				}
 			}

orchid.code-workspace

@@ -5,49 +5,66 @@
 		}
 	],
 	"settings": {
-		"editor.rulers": [
-			100 // Important; for accessibility reasons, code cannot be wider than 100ch
-		],
 		"[markdown]": {
+			// markdown denotes line breaks with trailing space
+			"diffEditor.ignoreTrimWhitespace": false,
+			// Disable editor gadgets in markdown
 			"editor.unicodeHighlight.ambiguousCharacters": false,
 			"editor.unicodeHighlight.invisibleCharacters": false,
-			"diffEditor.ignoreTrimWhitespace": false,
+			"editor.glyphMargin": false,
-			"editor.wordWrap": "bounded",
+			"editor.guides.indentation": false,
-			"editor.wordWrapColumn": 80,
+			"editor.lineNumbers": "off",
 			"editor.quickSuggestions": {
 				"comments": "off",
 				"strings": "off",
-				"other": "off"
+				"other": "off",
 			},
-			"editor.lineNumbers": "off",
-			"editor.glyphMargin": false,
 			"editor.rulers": [],
-			"editor.guides.indentation": false,
+			"editor.wordWrap": "bounded",
+			"editor.wordWrapColumn": 80,
+			// wrap lines as we go
 			"editor.formatOnType": true,
+      "editor.detectIndentation": false,
+      "editor.insertSpaces": false,
 		},
+		// Orchid is a human-made project
+		"chat.commandCenter.enabled": false,
+    // use spaces for indentation for Rust for now due to a rustfmt bug
+		"editor.tabSize": 2,
+		"editor.stickyTabStops": true,
+		"editor.detectIndentation": false,
+		"editor.insertSpaces": true,
+    // Important; for accessibility reasons, code cannot be wider than 100ch
+		"editor.rulers": [ 100 ],
 		"editor.formatOnSave": true,
-		"rust-analyzer.showUnlinkedFileNotification": false,
+		"git.confirmSync": false,
-		"rust-analyzer.checkOnSave": true,
+		"git.enableSmartCommit": true,
-		"rust-analyzer.check.command": "clippy",
+		"git.autofetch": true,
-		"rust-analyzer.rustfmt.extraArgs": [
+		"rust-analyzer.assist.emitMustUse": true,
-			"+nightly"
+		"rust-analyzer.assist.preferSelf": true,
-		],
 		"rust-analyzer.cargo.features": "all",
+		"rust-analyzer.check.command": "clippy",
 		"rust-analyzer.check.features": "all",
-		"files.associations": {
+		"rust-analyzer.checkOnSave": true,
-			"*.mjsd": "markdown"
+		"rust-analyzer.completion.fullFunctionSignatures.enable": true,
-		},
+		"rust-analyzer.completion.termSearch.enable": true,
+		"rust-analyzer.inlayHints.parameterHints.enable": false,
+		"rust-analyzer.inlayHints.typeHints.enable": false,
+		"rust-analyzer.rustfmt.extraArgs": [
+			"+nightly",
+		],
+		"rust-analyzer.showUnlinkedFileNotification": false,
 		"swissknife.notesEnabled": false,
 	},
 	"extensions": {
 		"recommendations": [
-			"maptz.regionfolder",
+			"fill-labs.dependi",
-			"tamasfe.even-better-toml",
-			"yzhang.markdown-all-in-one",
 			"gruntfuggly.todo-tree",
-			"vadimcn.vscode-lldb",
+			"maptz.regionfolder",
 			"rust-lang.rust-analyzer",
-			"fill-labs.dependi"
+			"tamasfe.even-better-toml",
+			"vadimcn.vscode-lldb",
+			"yzhang.markdown-all-in-one",
 		]
 	},
 }