Lesson 30-Deno Module Loader

Deno’s module system is one of its core features, offering modern ES module loading capabilities. This article delves into the working principles of Deno’s module loader, methods for custom implementation, and advanced application scenarios, helping developers master the core technologies of module loading.

Deno Module System Fundamentals

ESM Module Loading Process

Key Features:

  • URL-based module identifiers
  • Strict caching mechanism
  • Asynchronous loading process
  • Top-level await support

Built-in Module Loading Mechanism

Deno’s default module loader handles the following scenarios:

  • Local file modules (file:// protocol)
  • Remote HTTP modules (https:// protocol)
  • Data URL modules (data: protocol)
// Example: Loading modules with different protocols
import local from "./local.ts";          // Local file
import remote from "https://example.com/module.js"; // Remote HTTP
import data from "data:text/javascript,export const x=1;"; // Data URL

Custom Module Loader Implementation

Creating a Basic Loader

Implementation Principle:
Override import.meta.resolve and dynamic import() to implement custom loading logic.

// custom_loader.ts
const originalImport = globalThis.import;

globalThis.import = async (specifier: string) => {
  console.log(`[Custom Loader] Loading: ${specifier}`);

  // Custom resolve logic
  const resolved = await customResolve(specifier);

  // Custom fetch logic
  const source = await customFetch(resolved);

  // Use native import for processing
  return originalImport(source);
};

async function customResolve(specifier: string) {
  // Implement custom resolution logic
  if (specifier.startsWith("custom:")) {
    return `https://example.com/modules/${specifier.slice(7)}.js`;
  }
  return specifier;
}

async function customFetch(url: string) {
  // Implement custom fetch logic
  if (url.startsWith("https://example.com/")) {
    const response = await fetch(url);
    return await response.text();
  }
  throw new Error(`Unsupported URL: ${url}`);
}

Complete Loader Implementation

Production-Grade Loader Example:

// advanced_loader.ts
type ModuleCache = Map<string, { source: string; timestamp: number }>;

class CustomModuleLoader {
  private cache: ModuleCache = new Map();
  private resolver: (specifier: string) => Promise<string>;
  private fetcher: (url: string) => Promise<string>;

  constructor(
    resolver: (specifier: string) => Promise<string>,
    fetcher: (url: string) => Promise<string>
  ) {
    this.resolver = resolver;
    this.fetcher = fetcher;
  }

  async import(specifier: string) {
    // 1. Check cache
    if (this.cache.has(specifier)) {
      const entry = this.cache.get(specifier)!;
      if (this.isCacheValid(entry.timestamp)) {
        return this.evaluate(entry.source);
      }
      this.cache.delete(specifier);
    }

    // 2. Resolve module URL
    const resolvedUrl = await this.resolver(specifier);

    // 3. Fetch module source
    const source = await this.fetcher(resolvedUrl);

    // 4. Cache module
    this.cache.set(specifier, {
      source,
      timestamp: Date.now(),
    });

    // 5. Execute module
    return this.evaluate(source);
  }

  private isCacheValid(timestamp: number): boolean {
    return Date.now() - timestamp < 60 * 1000; // 1-minute cache
  }

  private async evaluate(source: string): Promise<any> {
    // Create sandbox environment
    const sandbox: any = {};
    const proxy = new Proxy(sandbox, {
      has: () => true, // Trick eval into thinking all variables exist
      get: (target, prop) => {
        if (prop === Symbol.unscopables) return undefined;
        return target[prop] || globalThis[prop];
      },
    });

    // Execute module code
    const fn = new Function("exports", "module", "require", "__filename", "__dirname", source);
    const module = { exports: {} };
    fn(module.exports, module, (spec: string) => this.import(spec), "", "");

    return module.exports;
  }
}

// Usage example
const loader = new CustomModuleLoader(
  async (specifier) => {
    if (specifier.startsWith("custom:")) {
      return `https://example.com/modules/${specifier.slice(7)}.js`;
    }
    return specifier;
  },
  async (url) => {
    if (url.startsWith("https://example.com/")) {
      const response = await fetch(url);
      return await response.text();
    }
    throw new Error(`Unsupported URL: ${url}`);
  }
);

// Dynamically load module
const module = await loader.import("custom:math");
console.log(module.add(1, 2));

Advanced Loader Patterns

Hot Module Replacement (HMR)

Implementation Principle:
Achieve module hot updates by listening for file changes and clearing the cache.

// hmr_loader.ts
class HMRLoader extends CustomModuleLoader {
  private watchers: Map<string, Deno.FsWatcher> = new Map();

  async import(specifier: string) {
    // Enable HMR in development environment
    if (Deno.env.get("DENO_ENV") === "development") {
      this.setupWatcher(specifier);
    }
    return super.import(specifier);
  }

  private setupWatcher(specifier: string) {
    if (this.watchers.has(specifier)) return;

    const filePath = this.resolveLocalPath(specifier);
    if (!filePath) return;

    const watcher = Deno.watchFs(filePath);
    this.watchers.set(specifier, watcher);

    (async () => {
      for await (const event of watcher) {
        if (event.kind === "modify") {
          console.log(`[HMR] Detected change: ${specifier}`);
          this.cache.delete(specifier); // Clear cache
        }
      }
    })();
  }

  private resolveLocalPath(specifier: string): string | null {
    if (specifier.startsWith("file://")) {
      return specifier.slice(7);
    }
    return null;
  }
}

Module Preloading

Preloading Strategy Implementation:

// preload_loader.ts
class PreloadLoader extends CustomModuleLoader {
  private preloadQueue: string[] = [];
  private isPreloading = false;

  async import(specifier: string) {
    // Add to preload queue
    if (!this.cache.has(specifier)) {
      this.preloadQueue.push(specifier);
      this.startPreloadIfNeeded();
    }

    // Load current module immediately
    return super.import(specifier);
  }

  private startPreloadIfNeeded() {
    if (this.isPreloading || this.preloadQueue.length === 0) return;

    this.isPreloading = true;
    const nextSpecifier = this.preloadQueue.shift()!;

    this.import(nextSpecifier).finally(() => {
      this.isPreloading = false;
      this.startPreloadIfNeeded();
    });
  }
}

Dependency Analysis Loader

Dependency Visualization Loader:

// dependency_loader.ts
class DependencyLoader extends CustomModuleLoader {
  private dependencyGraph: Map<string, string[]> = new Map();

  async import(specifier: string) {
    const dependencies = await this.analyzeDependencies(specifier);
    this.dependencyGraph.set(specifier, dependencies);

    return super.import(specifier);
  }

  private async analyzeDependencies(specifier: string) {
    // Implement dependency analysis logic
    // Simplified as returning an empty array
    return [];
  }

  printDependencyGraph() {
    console.log("Dependency Graph:");
    this.dependencyGraph.forEach((deps, module) => {
      console.log(`${module} -> ${deps.join(", ")}`);
    });
  }
}

Security-Enhanced Loader

Permission-Aware Loader

Permission-Checking Loader Implementation:

// secure_loader.ts
class SecureModuleLoader extends CustomModuleLoader {
  async import(specifier: string) {
    await this.checkPermissions(specifier);
    return super.import(specifier);
  }

  private async checkPermissions(specifier: string) {
    if (specifier.startsWith("file://")) {
      const filePath = specifier.slice(7);
      await this.checkFilePermission(filePath);
    } else if (specifier.startsWith("https://")) {
      await this.checkNetworkPermission(specifier);
    }
  }

  private async checkFilePermission(path: string) {
    try {
      const status = await Deno.permissions.query({ name: "read", path });
      if (status.state !== "granted") {
        throw new Error(`No file read permission: ${path}`);
      }
    } catch (err) {
      console.error("Permission check failed:", err);
      throw err;
    }
  }

  private async checkNetworkPermission(url: string) {
    try {
      const urlObj = new URL(url);
      const status = await Deno.permissions.query({ name: "net", host: urlObj.host });
      if (status.state !== "granted") {
        throw new Error(`No network access permission: ${urlObj.host}`);
      }
    } catch (err) {
      console.error("Permission check failed:", err);
      throw err;
    }
  }
}

Sandbox Isolation Loader

Secure Sandbox Implementation:

// sandbox_loader.ts
class SandboxModuleLoader extends CustomModuleLoader {
  private sandbox: any = {};

  async evaluate(source: string) {
    // Create fully isolated execution environment
    const proxy = new Proxy(this.sandbox, {
      has: () => true,
      get: (target, prop) => {
        if (prop === Symbol.unscopables) return undefined;
        return target[prop] || this.getSafeGlobal(prop);
      },
    });

    // Restrict available global objects
    const safeGlobals = {
      console: {
        log: console.log,
        error: console.error,
      },
      setTimeout: setTimeout,
      clearTimeout: clearTimeout,
    };

    const fn = new Function(
      "exports",
      "module",
      "require",
      "__filename",
      "__dirname",
      source
    );

    const module = { exports: {} };
    fn(module.exports, module, this.import.bind(this), "", "");

    return module.exports;
  }

  private getSafeGlobal(prop: string) {
    if (prop in safeGlobals) {
      return safeGlobals[prop];
    }
    throw new Error(`Access to global object prohibited: ${prop}`);
  }
}

Performance-Optimized Loader

Cache Strategy Optimization

Multi-Level Cache Implementation:

// cache_loader.ts
class CacheModuleLoader extends CustomModuleLoader {
  private memoryCache: ModuleCache = new Map();
  private diskCache: DiskCache;

  constructor(
    resolver: (specifier: string) => Promise<string>,
    fetcher: (url: string) => Promise<string>,
    cacheDir: string
  ) {
    super(resolver, fetcher);
    this.diskCache = new DiskCache(cacheDir);
  }

  async import(specifier: string) {
    // 1. Check memory cache
    if (this.memoryCache.has(specifier)) {
      return this.evaluate(this.memoryCache.get(specifier)!.source);
    }

    // 2. Check disk cache
    const diskCached = await this.diskCache.get(specifier);
    if (diskCached) {
      this.memoryCache.set(specifier, diskCached);
      return this.evaluate(diskCached.source);
    }

    // 3. Fetch new module
    const resolvedUrl = await this.resolver(specifier);
    const source = await this.fetcher(resolvedUrl);

    // 4. Update cache
    const cacheEntry = { source, timestamp: Date.now() };
    this.memoryCache.set(specifier, cacheEntry);
    await this.diskCache.set(specifier, cacheEntry);

    return this.evaluate(source);
  }
}

class DiskCache {
  constructor(private dir: string) {
    Deno.mkdirSync(dir, { recursive: true });
  }

  async get(specifier: string): Promise<{ source: string; timestamp: number } | null> {
    const file = this.getCacheFile(specifier);
    try {
      const content = await Deno.readTextFile(file);
      const { source, timestamp } = JSON.parse(content);
      if (this.isCacheValid(timestamp)) {
        return { source, timestamp };
      }
      await Deno.remove(file);
      return null;
    } catch (err) {
      return null;
    }
  }

  async set(specifier: string, entry: { source: string; timestamp: number }) {
    const file = this.getCacheFile(specifier);
    const content = JSON.stringify(entry);
    await Deno.writeTextFile(file, content);
  }

  private getCacheFile(specifier: string): string {
    const hash = this.hashSpecifier(specifier);
    return `${this.dir}/${hash}.cache`;
  }

  private hashSpecifier(specifier: string): string {
    // Simple hash implementation
    let hash = 0;
    for (let i = 0; i < specifier.length; i++) {
      hash = ((hash << 5) - hash) + specifier.charCodeAt(i);
      hash |= 0;
    }
    return hash.toString(16);
  }

  private isCacheValid(timestamp: number): boolean {
    return Date.now() - timestamp < 5 * 60 * 1000; // 5-minute cache
  }
}

Parallel Loading Optimization

Concurrency Control Loader:

// parallel_loader.ts
class ParallelModuleLoader extends CustomModuleLoader {
  private concurrency = 3;
  private queue: Array<() => Promise<void>> = [];
  private activeCount = 0;

  async import(specifier: string) {
    return new Promise((resolve, reject) => {
      const task = async () => {
        try {
          this.activeCount++;
          const result = await super.import(specifier);
          resolve(result);
        } catch (err) {
          reject(err);
        } finally {
          this.activeCount--;
          this.next();
        }
      };

      this.queue.push(task);
      this.next();
    });
  }

  private next() {
    if (this.activeCount < this.concurrency && this.queue.length > 0) {
      const task = this.queue.shift()!;
      task();
    }
  }
}

Practical Application Cases

Plugin System Implementation

Plugin Architecture Based on Custom Loader:

// plugin_system.ts
class PluginSystem {
  private loader: CustomModuleLoader;
  private plugins: Map<string, any> = new Map();

  constructor(loader: CustomModuleLoader) {
    this.loader = loader;
  }

  async loadPlugin(name: string) {
    if (this.plugins.has(name)) {
      return this.plugins.get(name);
    }

    const plugin = await this.loader.import(`plugins/${name}.ts`);
    this.plugins.set(name, plugin);
    return plugin;
  }

  async executePlugin(name: string, ...args: any[]) {
    const plugin = await this.loadPlugin(name);
    if (typeof plugin.execute === "function") {
      return plugin.execute(...args);
    }
    throw new Error(`Plugin ${name} does not implement execute method`);
  }
}

// Usage example
const loader = new SecureModuleLoader(/* Permission check logic */);
const pluginSystem = new PluginSystem(loader);

// Dynamically load and execute plugin
await pluginSystem.executePlugin("analytics", { track: true });

Multi-Environment Module Loading

Environment-Aware Loader:

// env_aware_loader.ts
class EnvironmentAwareLoader extends CustomModuleLoader {
  private env: string;

  constructor(
    env: string,
    resolver: (specifier: string) => Promise<string>,
    fetcher: (url: string) => Promise<string>
  ) {
    super(resolver, fetcher);
    this.env = env;
  }

  async resolve(specifier: string) {
    if (specifier.startsWith("env:")) {
      const envSpecifier = specifier.slice(4);
      return `./env/${this.env}/${envSpecifier}.ts`;
    }
    return specifier;
  }
}

// Usage example
const loader = new EnvironmentAwareLoader(
  Deno.env.get("DENO_ENV") || "development",
  async (specifier) => {
    if (specifier.startsWith("env:")) {
      return this.resolve(specifier);
    }
    return specifier;
  },
  async (url) => {
    // Implement fetch logic
    return fetch(url).then(res => res.text());
  }
);

// Load environment-specific module
const config = await loader.import("env:database");
Course: Mastering Core Concepts of Bun.js, Deno.js, and Winter.js
Share your love

Trending now

Exploring Web 3D Core Technologies Development
WebRTC Advanced Guide：Media, Transport & Architecture
WebSocket Deep Guide：Building High-Performance Realtime Systems
Micro Frontends & Serverless: Modular Web Development