App Development with Siri and AI
Hi, Future App DeveloperSee you on the 25th at the Trottier Building
Event details

McHacks · Interactive session

New to app development?

Don’t worry if you don’t know how to code at all. Come with a vision for your app: this talk will show you how to build it and get it into real people’s hands.

Illustration of the app-development workshop: a laptop with code, a simulator and a phone

Before we start

Build something you care about.

A complete app, not just an MVP. AI makes the rest doable.

  1. Useful.
  2. Complete.
  3. Yours.

What we’ll do together

Build an app. Add AI. Keep improving.

Then: ship it to real phones, and look after it.

The example app

Meet Commit, the app we’ll follow

Watch Siri adjust a workout

Commit Today screen
TodaySmall daily habits to tick off
Commit Training screen
TrainingThis week’s plan, one session a day
Commit Check-in screen
Check-inMood, sleep, calm and what’s sore
Commit Workout screen
WorkoutAdapts to your check-in, with videos
Commit Siri screen
SiriToday’s workout without opening the app

Built with Daniela Suarez, founder of Elovlo Inc.

The words we’ll use

Eight words for adding AI to an app

In the order the talk explains them.

  1. 1Foundation ModelsApple’s model, from Swift
  2. 2On-device modelIt runs on the phone
  3. 3ContextInstructions + facts you give
  4. 4ToolsFunctions it can call
  5. 5Typed answersSwift fields, not free text
  6. 6App IntentsActions Siri can run
  7. 7Image inputRead a photo or a sign
  8. 8EvaluationsTests that show it works

On the website too

Private Cloud ComputeCore AI and Core MLCreate MLMLXVisionSpotlight and app memoryWriting ToolsGenmojiApple’s model in Terminal

Part 1 · 17 minutes

Build a real app

Two real apps: Commit, a wellness app on TestFlight, and Passage, a Montréal métro app we built.

  1. Opening and agenda
  2. Why build an app?
  3. What an app can do
  4. Databases
Commit
Passage

Overview

Build the app. Then add useful AI.

Start with app development, or explore how to add AI features to your app.

Build your app using Swift and SwiftUI in Xcode. Then add useful AI features to the working app with selected context and tools.Build the appAdd useful AISwift XcodeAIContext + toolsYour app first. Then useful AI.
Build your app first. Then add useful AI.
Swipe to explore the diagram.

Part 2 · Add useful AI

Why add AI locally?

Log sore legs without leaving the Home Screen. Commit uses the saved check-in to adjust your workout.

  1. Part 1Build a real app
  2. Part 2Why add AI locally?Your own modelWhere the model runs
  3. ThenShip it
Commit

Overview

AI technologies for your app

Choose a framework for your feature. Open a technology to see how it works.

Understand a question, perform an app action, look at an image and check the result. Choose Foundation Models, tools and App Intents, Vision or evaluations to fit the feature.What should the feature do?UnderstandFoundation ModelsActTools + App IntentsLookVisionCheckEvaluationsPick the framework after the task.
Choose the feature first, then the framework.
Swipe to explore the diagram.

Core AI

New in iOS 27: convert a PyTorch model once, then run it on the phone’s CPU, GPU and Neural Engine. Your data stays private and the cloud bill is zero. Open the slide

Foundation Models framework

Use Apple’s language model from Swift. It runs on the iPhone, so it’s private, free and offline, and it can return typed Swift values instead of loose text. For longer reasoning, one line switches to Private Cloud Compute*, Apple’s bigger cloud model. Open the slide

Vision

Name what’s in a photo, read its text, and find faces, body poses and barcodes. Vision’s ready-made requests need no training and run on the device. Open the slide

Speech

Turn speech into text with SpeechAnalyzer and SpeechTranscriber. It runs on the device, so you can caption a lecture without sending audio to a server. The model downloads the first time you use it. Open the slide

Machine-learning APIs

Natural Language, Translation and Sound Analysis take a few lines each. Tag names, places and mood, translate text, or name one of 300+ sounds. Open the slide

Core ML

Train a model on your Mac with Create ML, or convert one from PyTorch, TensorFlow or scikit-learn with coremltools. Drop the file into Xcode and it runs on the CPU, GPU and Neural Engine. Open the slide

Metal

Core ML and MLX already use the GPU through Metal. Write your own kernels, or use Metal 4’s tensors, when no framework has the operation you need. Measure first. Open the slide

MLX

Run open models on your Mac with MLX, Apple’s open-source stack. Chat with a 3B model, fine-tune it, or serve it to Xcode and your app through a local, OpenAI-compatible server. Open the slide

Part 2 · The words we’ll use

Eight words for adding AI to an app

Ask

Part 2 uses eight terms. Which do you already know?Which of these have you heard before?

Eight terms for Part 2, in the order the slides explain them: Foundation Models, Apple's model from Swift; on-device model, it runs on the phone; context, the instructions and facts you give; tools, functions it can call; typed answers, Swift fields instead of free text; App Intents, actions Siri can run; image input, reading a photo or a sign; evaluations, tests that show it works. The website also covers Private Cloud Compute, Core AI and Core ML, Create ML, MLX, Vision, Spotlight and app memory, Writing Tools, Genmoji and Apple's model in Terminal.1Foundation ModelsApple's model, from Swift2On-device modelIt runs on the phone3ContextInstructions + facts you give4ToolsFunctions it can call5Typed answersSwift fields, not free text6App IntentsActions Siri can run7Image inputRead a photo or a sign8EvaluationsTests that show it worksOn the website tooPrivate Cloud Compute · Core AI and Core ML · Create ML · MLX · VisionSpotlight and app memory · Writing Tools · Genmoji · Apple's model in Terminal
Eight words, in the order the slides explain them.
Swipe to explore the diagram.
Read the full explanation

In the order we explain them

Foundation ModelsApple’s framework for using its language model from Swift.
On-device modelThe model runs on the phone: private, free per request, works offline.
ContextThe instructions and facts you give the model.
ToolsFunctions the model can call, such as weather, maps or your server.
Typed answersGuided generation: the model fills Swift fields instead of free text.
App IntentsActions Siri and Shortcuts can run in your app.
Image inputReading a photo or a sign.
EvaluationsTests that show the feature works.

Also on this website: Private Cloud Compute, Core AI and Core ML, Create ML, MLX, Vision, Spotlight and app memory, Writing Tools, Genmoji and Apple’s model in Terminal.

ModelFoundation Models, on the device.
Feed itContext and tools.
Use itTyped answers, App Intents, images, evaluations.

Follow along

Each Part 2 slide names its word at the top.

Relate

Eight words, one app. Passage shows each one.

Add AI features to your app

Start with one useful task. Choose a model, give it app context and tools, then test the result.

Two integration starting points

Add AI features to your app

iOS 27, Xcode 27 and macOS 27

Build your app

Take five to six minutes to learn why we use Swift, what to install, how to improve code with a language model, and how an app reaches your phone and the App Store. Then explore AI through Passage.

Concept illustration of the app-development workshop: laptop, simulator and phone

Chapter 1 of 4

Start here

First, the basics of building and sharing an app. Then Passage shows how models, useful context and actions fit together.

Today

Build an app. Add AI. Keep improving.

Build your app, connect useful system features, add an AI feature, test the result and keep improving something you care about.BuildWidgetAlertFeaturesAITestYoursBuild something useful to you.
Build. Add features. Add AI. Test. Improve.
Swipe to explore the diagram.

Part 1 · 17 minutes

Build a real app

With Commit, a wellness app already on TestFlight

  1. Opening and agenda
  2. Why build an app?
  3. What an app can do
  4. Databases

Part 2 · 14 minutes

Add useful AI

Commit · Siri and on-device personalization

  1. Why add AI locally?
  2. Train or bring your own model
  3. Where the model runs
Part 1Part 2Ship it

Then: ship it, then look after it · 1.5 min

newAdd AI features to your appModels, app context, tools and evaluation: a practical path into AI integration.Open → New to app development?Swift, setup, coding with AI, GitHub, backend choices and your first release.Open →
1 · Build the app

Start with one useful part

Learn the setup, code, data and release steps through your own app.

2 · Add an AI feature

Give it context and tools

Connect a model, inspect its sources and check the action it proposes.

3 · Keep improving

Go beyond the first demo

Test it with people, fix what breaks and make the work useful.

Your idea. Your app. Your decisions.

We start with a 5½-minute app-development primer. Everything else, including the parts we skip today, is on this page to read later. Sources appear under each topic.

App development in 5½ minutesSwift, Xcode, iteration, GitHub, backend choices and release

App basics

Why are we building with Swift?

Ask

One app for Android and iOS, or a direct route into Apple’s newest AI?Which path fits our app?

Two ways to build: React Native shares one JavaScript codebase across iOS and Android but reaches native features through bridges and gets new Apple features later; Swift and SwiftUI are native, fast and get the newest Apple features such as Health, Siri and widgets, but run only on Apple devices and need a Mac.Your appReact NativeOne codebaseiOS + AndroidJavaScript skillsBridges to nativeApple features laterSwift + SwiftUINative speedNewest Apple featuresHealth, Siri, widgetsApple onlyNeeds a Mac
Choose the app path, then connect the APIs it needs.
Swipe to explore the diagram.
Read the full explanation
Across platforms

React Native

A framework using React and JavaScript or TypeScript to build Android and iOS apps.

Our route today

Swift + SwiftUI

Swift is the language; SwiftUI builds the interface. We target Apple devices.

Foundation Models provides a native Swift API. Swift gives this workshop direct access to the AI features we will demonstrate.

Choose a pathShared app code or native Apple APIs.
Use the building blocksConnect the features your app needs.
Check availabilityOS, hardware and model support.

What the update unlocks

New system APIs and AI capabilities become building blocks for our own apps.

Check the feature’s OS, hardware and model availability. An OS update alone does not enable every AI feature.

Relate

React Native can connect to native APIs through modules. Swift keeps our Apple-focused workshop direct.

React Native · Apple Foundation Models

App basics

Write Swift in Xcode. Run your app.

Ask

What runs the app before it ever reaches your iPhone?What do we need to run it?

Xcode’s agent, step by step

  1. Plan
  2. Code
  3. Build + test
  4. Preview
  5. You review

Your choice of agentClaudeChatGPTGeminior a model on your Mac

Siri on an iPhone: My legs are sore in commit. Reply: Turn on Use Commit with Siri in Commit Settings first.
Real iPhone
Test every feature on a real phone

The Simulator can’t catch everything. On my iPhone, Siri said: Turn on Use Commit with Siri in Commit Settings first.

Settings, permissions, Siri and notifications only show their real behaviour on the device.

Real screenshots: Xcode with the Passage project open, and the Simulator running Passage on an iPhone 18 Pro, side by side on a Mac. Then the same Passage screen on your own iPhone, to check the feature for real. Xcode's agent works step by step: plan, code, build and test, preview, then you review. Your choice of agent: Claude, Codex or Gemini, or a model on your Mac.Xcode File Edit View Find Navigate Editor Product Debug123Xcode + Simulator on your MacBuild in Swift + SwiftUI, then try it on the MacYour phoneCheck it for real
Try it in the Simulator first, then on your phone.
Swipe to explore the diagram.
Read the full explanation
A compatible MacCheck the macOS requirement for your Xcode version.
Free disk spaceXcode, iOS runtimes, downloaded models and build files all take space.
RAM + computing powerThe editor, build tools and Simulator share your Mac’s resources. Start with one simulator.
Prepare XcodeInstall the SDK and Simulator runtime.
Open one projectUse Swift + SwiftUI or the prepared app.
Choose a device · RunSimulator uses your Mac’s resources.

Install → choose → run

  1. Install Xcode from the Mac App Store.
  2. In Xcode Settings → Components, download iOS support and a simulator runtime.
  3. Create an iOS App project with Swift and SwiftUI, or open the prepared project.
  4. Select an iPhone simulator as the run destination. Press Run.
Relate

Xcode is the workshop. Simulator is a test phone running on your Mac; it uses the Mac’s resources.

Get Xcode · Install components

App basics

What only an app can do

Ask

What can an iPhone app reach that a website cannot?What can trigger an app action?

What only an app can do. Inside the phone: haptics you can feel, sensors for steps, tilt and heading, the camera and AR to place things in 3D, and widgets to glance without opening the app. Around the phone: Bluetooth for robots, drones and sensors, NFC to tap a tag or a card, Ultra Wideband for distance and direction, and Home and Matter for lights, locks and plugs. Safari on iPhone cannot use Bluetooth, NFC or Ultra Wideband; apps can.Inside the phoneHapticsFeel a tap patternSensorsSteps, tilt, headingCamera + ARPlace things in 3DWidgetsGlance without openingThings around itBluetoothRobots, drones, sensorsNFCTap a tag or a cardUltra WidebandDistance + directionHome + MatterLights, locks, plugsYour appSafari on iPhone can’t use Bluetooth, NFC or Ultra Wideband. Apps can.
The phone’s own hardware, and the devices around it. No AI needed.
Swipe to explore the diagram.
Read the full explanation

The phone itself, and the things around it

HapticsCore Haptics plays tap patterns people feel: a confirmation, a countdown, a gentle nudge.
SensorsCore Motion reads steps, tilt, heading and altitude, with the person’s permission.
Camera + ARARKit places and measures things in the room; LiDAR on Pro iPhones adds depth.
Widgets + Live ActivitiesWidgetKit and ActivityKit show the app on the Home Screen, the Lock Screen and the Dynamic Island, without opening it.
BluetoothCore Bluetooth talks to robots, drones, heart-rate straps and sensors you build; AccessorySetupKit pairs them with one system sheet.
NFCCore NFC reads tags and supported cards when the phone touches them.
Ultra WidebandNearby Interaction gives distance and direction to another iPhone, an Apple Watch or a compatible accessory.
Home + MatterHomeKit controls the lights, locks and plugs in the person’s Home.
And moreApple Watch apps, CarPlay, game controllers, and Wi‑Fi Aware for direct phone-to-phone links.

Safari on iPhone supports neither Web Bluetooth nor Web NFC, and the web has no Ultra Wideband API. None of these features needs AI: the phone itself is the feature.

InputAn incoming call in a calling app, or a button tap.
ActionRun the app code for that event.
Output → next inputA saved value can feed another action.

Pick one that fits your idea

  1. Name the moment: what should the person feel, see or connect?
  2. Find the framework: Core Haptics, Core Motion, ARKit, WidgetKit, Core Bluetooth, Core NFC, Nearby Interaction or HomeKit.
  3. Test on a real iPhone: Bluetooth, NFC and Ultra Wideband need the phone’s radios.
Relate

None of these need AI. The phone itself is the feature.

Core Haptics · Core Bluetooth · AccessorySetupKit · Core NFC · Nearby Interaction · WidgetKit · HomeKit · Wi‑Fi Aware · Web NFC support · Web Bluetooth support

App basics · 45 seconds

Build one small change at a time

Ask

The coding assistant says “done.” How do you know the change works?What is one useful change?

Ask for one visible change: show when a note is saved. Compare the old and changed app screen, build and inspect, then use the result to revise. The saved note stays unchanged.Show when a note is savedOne requestBeforeMy noteSave buttonAfterMy noteSaved ✓Build · inspect · reviseSame note. Clearer feedback.
Illustrated change: inspect the result, then improve the next request.
Swipe to explore the diagram.

TestFlight · What to test

“test the workout plans”

One note per build, for the people testing it.

Build 1
Build 2: same screen, improved
Build 3: swipe the weekly plan
Read the full explanation
  1. DescribeOne visible change
  2. GenerateLet the agent edit
  3. Build + runRead errors; try it
  4. Inspect + repeatKeep a checkpoint
One bounded requestShow when a note is saved.
Use existing dataKeep saving and storage unchanged.
Build · inspect · checkpointCheck the result before keeping it.

A useful request

“Show a clear confirmation when a note is saved. Keep the existing data and saving logic. Build it and tell me how to check the result.”

Give the next prompt evidence

Share the exact build error, screen or failed action. Review the changes, try the result, then save a working checkpoint in Git.

The coding assistant builds the app. The AI inside the app is a separate component.

Relate

Learn the layers as you build: app logic → data → model → actions → evaluation. Today we use an existing model.

Databases

Does this app change need a database change?

Ask

If we change what people see, do we also need to change what we save?Commit asks if you track your cycle. App change, or database change too?

Recorded in Commit

Who needs the answer?
Only this phone → app change.
Another device or the server → database change too.

One change: add a cycle question to the app. As an app change, only this phone uses the answer: the screen and this phone's workout suggestions change, and the database does not. As an app and database change, the answer goes into the synced profile as an optional field, so every device, the server's plans and older app versions are affected; the next slide shows what the database checks before it saves.App changeOnly this phone uses itApp + database changeEvery device and the server use itPlanCycleoptionalSynced profileNameCycle · optionalChanges: the screen, this phone’sworkout suggestionsChanges: every device, the server’splans, older app versionsNo database changeNext slide: what the database checks
One change: add a cycle question to the app. As an app change, only this phone uses the answer: the screen and this phone's workout suggestions change, and the database does not. As an app and database change, the answer goes into the synced profile as an optional field, so every device, the server's plans and older app versions are affected; the next slide shows what the database checks before it saves.App changeOnly this phone uses itApp + database changeEvery device and the server use itPlanCycleoptionalSynced profileNameCycle · optionalChanges: the screen, this phone’sworkout suggestionsChanges: every device, the server’splans, older app versionsNo database changeNext slide: what the database checks
The app can change what it shows. Saving new information across devices also changes the data it stores.
Swipe to explore the diagram.
How we track and release a code change
BranchA separate line of work for the next change.
CommitA saved checkpoint. GitHub hosts the shared history.
Pull requestReview and test the change before combining it.
Release tagA fixed label for the exact source used to build a version.
Existing usersStart with the 500 profiles already in use.
New usersLet people leave the field unset.
Both versionsTest reading and saving with each build.

A release-first convention

  1. Keep main at the publicly released app’s source.
  2. Iterate and test on a work branch.
  3. Freeze a release candidate; record its commit, version and build number.
  4. After approval and release, advance main to that source and tag it.
Relate

Keep the published edition on one shelf and the next draft on another. You can keep writing during review.

This is one Git convention, chosen for a clear shipped-version reference. Apple reviews an uploaded build, not your GitHub branch. Approval can precede a manual release.

App Store Connect: builds, review and release

App basics

Where should app records and files live?

Ask

Does saving a favourite, a note or a photo always need a server?What needs to be saved or shared?

Keep records and files on the phone first: health data, settings and drafts need no account and work offline. Only when data must reach another device, other people or a backup, add a backend for sign-in, a database, file storage and access rules, such as Firebase, Supabase, CloudKit or AWS. Commit keeps health data on the phone and syncs sign-in, the plan and check-ins through Firebase.On the phoneHealth dataSettingsDraftsNo account neededWorks offline1keeps health data on the phoneNeed to share?another device other people backup2Your backendSign-inDatabaseFile storageAccess rulesFirebaseSupabaseCloudKitAWS3syncs sign-in, the plan and check-insStart on the phone. Add a backend only to share.Commit
Choose the data first; add sharing when it is needed.
Swipe to explore the diagram.
Read the full explanation

On this phone

SwiftData or local files can store the app’s records. Secrets belong in the Keychain.

Shared or synced data

Firebase is one option: Authentication handles sign-in, Firestore stores records and Cloud Storage holds files. Compare Supabase, AWS and CloudKit below.

Firebase works with Apple, Android and web apps. It sits outside Apple’s ecosystem; it is an option when the product needs a backend.

Start on the deviceNo backend needed for local saves.
Records or files?Database entries or photos and PDFs.
Choose a backendCompare access rules, cost and setup.

Connect a backend deliberately

  1. Decide what stays local, what becomes shared, and whether it is a record or file.
  2. Choose the necessary backend services, then add their supported app configuration and SDK.
  3. Configure identity, per-user record and file access, recovery and usage controls.

We explain the option here. A private app can keep notes on the device without a backend or upload.

Relate

Start with a notebook in your pocket. Add a shared notebook when people or devices need to share.

Firebase for Apple apps · Firestore access rules

Compare backends, files, AWS and costs
Backend options: records, files and tradeoffs
OptionStores / connectsUseful benefitTradeoff
FirebaseAuthentication; Firestore records; Cloud Storage files.Mobile SDKs, realtime records and offline caching.Document/query model; database and file rules; separate usage charges.
SupabaseAuth; Postgres records; Storage file buckets.SQL tables and relationships with managed APIs.Design schemas, row-level access and file policies; plan record and file backups.
AWS with AmplifyOne route uses Cognito, AppSync, DynamoDB and S3.Connect an app to AWS services and detailed access controls.More service/permission choices; the underlying services have separate costs.
Apple CloudKitRecords and CKAsset attachments; private, shared and public databases.An Apple-centered app can use the person’s iCloud account.iCloud/account and quota constraints; a web API exists, but it is not the same cross-platform setup as the other options.

Firebase files · Supabase Storage · Amplify services · CloudKit

Can we save notes, photos, audio or PDFs?

Yes, with an appropriate storage service and supported limits. Store fields such as owner, item ID, note text and file path in a database. Store large attachments in local files or object storage, with a database record pointing to each file. For example, an item record can reference its attached photo in S3. Storing a file does not mean the selected AI model can interpret its format.

What connecting an Apple app to AWS means

Amplify Swift can connect the app to AWS resources. In one architecture, Cognito handles identity, AppSync exposes the API, DynamoDB holds records and S3 holds photos or documents. Choose only the services needed. Use per-user authorization and temporary access or signed links where appropriate; do not distribute account-wide AWS credentials. Connect existing AWS resources.

Free tiers have limits

Storage is not unlimited and free by default. Stored bytes, downloads, requests, database operations, compute and authentication features can have different meters. AWS offers depend on the account, credits and plan. Firebase file storage requires Blaze and billing even where qualifying usage has a no-cost allowance; Firestore database quotas are separate. Supabase also separates database, file and transfer quotas. CloudKit private data uses the person’s iCloud quota. These conditions were checked on 24 October 2026; consult the live pricing pages before choosing.

S3 pricing · AWS Free Tier · Firebase storage billing · Supabase plans

Local saving, sync and backup are different. Add shared storage when the product needs it, configure both record and file permissions, test allowed and denied access, and plan recovery. A budget alert is not a universal spending cap. Passage’s demonstration still saves its favourites and notes locally; no backend account or upload is needed. An AWS backend can be used with an on-device model: storage does not choose or train the model.

Next · PrivacyWho holds the data?Your phone, the company, or a model provider →

App basics

Your app can connect to Apple services

Ask

Which familiar phone feature could make your app more useful?What should the app be allowed to do?

Your app asks, the person sees the system prompt and taps Allow or Don’t Allow, and only then does a capability such as Siri and Shortcuts, notifications, HealthKit or Apple Pay work. Each still needs its own setup and real-device checks.Your appasks“Commit” Would Like to Send You NotificationsA reminder for today’s check-inDon’t AllowAllowSiri + ShortcutsNotificationsHealthKitCheckout123Ask when it helps; the app still works if they say no.
Capabilities connect through the user’s choices.
Swipe to explore the diagram.
Reminders
Apple Health

Recorded in Commit

Read the full explanation
Siri + ShortcutsApp Intents exposes specific app actions to the system.
Mail + MessagesPresent a composer; the person reviews and sends.
Apple HealthHealthKit requests access to specific health-data types.
Apple PayUse a familiar checkout for eligible goods and services.
Choose a capabilityUse the relevant native API.
Request permissionThe user controls data and actions.
Test on a real deviceSetup and hardware checks still apply.

Built-in connections, explicit choices

Apple provides native APIs and familiar system screens. Add the required capabilities, request permission where needed, and test on a real device.

Websites can use Apple Pay too, with web and merchant setup. Native integration can reduce custom work; it still needs configuration.

Relate

Use the phone’s existing building blocks. The user decides what to share and which actions to take.

Apple Pay in apps and on the web · Mail and Messages · HealthKit permissions

App basics

The right reminder, at the right moment

Commit
Ask

What helps someone without making them reopen the app?What helps people between visits?

A local notification arrives at the right moment, one tap opens Commit on today's session. No AI is required.COMMIT · nowYour 31-min sessionis ready · tap to start1One tap2Straight totoday’s sessionCommit’s real screen3A local notification and a link into the right screen. No AI required.
A local notification and a link into the right screen.
Swipe to explore the diagram.
Daily boost

Recorded in Commit

Read the full explanation

Connect useful system experiences before integrating your own model: local reminders or remote push, widgets and Live Activities, and accessible feedback. Add them when they serve a concrete need.

Timely alertsLocal reminder or server push.
At a glanceWidgets and Live Activities.
Comfortable useAccessible text, VoiceOver and haptics.

Choose one experience

Ask for permissions at the relevant moment. Test supported devices, delivery, empty states and accessibility.

Relate

Be useful at the right moment, with the person in control.

Options, requirements and Apple sources

These features can improve an app before it integrates its own model. Choose a concrete benefit, use the matching system API and request access only when the feature needs it.

The study-app example: a student has a Statistics exam tomorrow and chooses to revise today at 6 PM. A local notification invites them to review five saved practice questions. Tapping it opens that practice set directly at question one. The app requests notification access when the student enables the reminder, schedules the alert and handles its tap with ordinary app navigation. This example does not require an AI model. An exam-countdown widget, adaptable text, VoiceOver labels and visible confirmation can improve the same app further.

Additional app-experience options
FeatureUseful benefitSetup or limit
App IntentsSelected actions in Siri and Shortcuts.Expose entities, queries and actions deliberately.
Local notificationAn app-created reminder.Authorization plus a scheduled trigger; no push server required.
Remote pushAn alert caused by a server event.Provider server → APNs → device; registration and setup.
WidgetsUseful information outside the app.WidgetKit timelines; system-controlled update budgets.
Live ActivitiesProgress on the Lock Screen or Dynamic Island.ActivityKit state, updated by the app or its server.
HapticsTactile confirmation of an interaction.Supported devices and settings; retain a visible cue.
AccessibilityAdaptable text and navigation.Dynamic Type, VoiceOver, clear labels and adaptable layouts.
Apple RemindersAn entry in the person's Reminders database.EventKit and Reminders access; distinct from an app notification.
Share sheetShare content through familiar system choices.Supply content; the person selects a destination.
Universal linksOpen the relevant app content from a link.Associated domains and a matching website association.

Options for your own app, not claims that every feature is implemented in Passage. Sources checked 25 October 2026.

App basics

From Simulator to someone else’s phone

Commit
Ask

It works on the Mac. What do we check before sharing it?Who should try it before release?

From your Mac to everyone, in five steps: try the app in the Simulator on your Mac (the same Xcode and Simulator as before), run it on your own phone with Developer Mode, optionally let testers try it in TestFlight, Apple checks the build in App Review (it works, privacy, the guidelines) and approves it, then you choose when to release it. A working demo is the beginning.It worksPrivacyGuidelinesApprovedGet12345SimulatorOn your MacYour phoneDev ModeTestFlighttesters, optionalApp ReviewApple checks itReleaseYou choose whenA working demo is the beginning.
Test the app before wider release; TestFlight is optional.
Swipe to explore the diagram.
Read the full explanation
  1. SimulatorFast iteration on Mac
  2. Your iPhoneReal hardware + permissions
  3. TestFlightBeta testing + feedback
  4. App ReviewThen App Store release
  1. 1SimulatorOn your Mac
  2. 2Your phoneDeveloper Mode
  3. 3TestFlightTesters, optional
  4. 4App ReviewApple checks it
  5. 5ReleaseYou choose when

A working demo is the beginning.

Check your devicePair, sign and enable Developer Mode.
Optional betaUse TestFlight for feedback.
Review, then releaseApproval and publication are separate.

Own phone: pair, sign, enable Developer Mode and Run.
Sharing: upload a build, collect beta feedback, then submit for review.

Device setup and submission details

Get it onto your own phone

Pair the iPhone with your Mac. Add your Apple Account and signing team in Xcode, enable Developer Mode on the phone, select it as the run destination and press Run.

Share a build, then submit

Archive in Xcode and upload to App Store Connect. Testers install TestFlight and accept your invitation. Collect feedback, fix issues, then submit a build with its listing, privacy details and review information.

Relate

Your test bench → your phone → beta testers → public release. Return to iteration whenever something fails.

A free Apple Account supports personal device testing with limits. TestFlight and App Store distribution need Developer Program membership. TestFlight is optional; external beta review is separate from App Store review.

Run on a device · TestFlight · Submission workflow

Passage on stage: choose, improve and inspect

  1. 1IdeaSend, then vote
  2. 2BuildPrompt into Xcode
  3. 3AIOn-device model
  4. 4ActionAn App Intent
  5. 5ModelBuilt-in ML
  6. 6MacCLI & Python companions

In the timed talkIf time allows

Foundation Models

Don’t change the model. Give it context.

Ask

With Swift and the Foundation Models framework, the model is already on the phone. What does your app add?What does the model need from you?

What can be context for the same on-device model: instructions you write (role, rules, format); your app's data (saved stations, history); what's on screen (the station in view); the person (their words and preferences); the phone, if allowed (Calendar, Health, Photos); servers through tools (weather, Maps, your API); and the conversation (earlier turns, kept short). The result is an answer that fits this person at this moment. Same model for every app; your context makes it yours.InstructionsRole, rules, formatYour app's dataSaved stations, historyWhat's on screenThe station in viewThe personWords and preferencesThe phone, if allowedCalendar, Health, PhotosServers, via toolsWeather, Maps, your APIThe conversationEarlier turns, kept shortApple'smodelAn answer that fitsthis person, this momentSame model for every app. Your context makes it yours.
What can be context: seven sources, one model.
Swipe to explore the diagram.
Read the full explanation

The model ships with the phone

No training, no keyOn supported devices with Apple Intelligence, the Foundation Models framework gives your Swift code Apple’s on-device model. You don’t train it, host it or pay per request.
From your appRecords the app already has: Passage’s saved stations and the métro map.
From the personWhat they type or say, and their choices: “McGill to Jean-Talon, fewer transfers.”
From serversTools fetch current facts, such as weather or maps. Those queries leave the device.

The app still checks what the model proposes before it acts on it.

The model is readyOn the phone, no training, no key.
Your app + the personRecords and what they ask.
Servers, through toolsCurrent facts; those queries leave.

Give it context

  1. Instructions: what the feature is for.
  2. Facts from the app and the person, in the prompt.
  3. Tools for anything current or outside the phone.
Relate

Same model for every app. Your context makes it yours.

Foundation Models · Tool calling

Context

AI alone isn’t great: context makes it useful

Ask

You tell Siri: “My legs are sore in Commit.” What does it need to know?

Your turn
Commit's App Intent tells Siri to pick one body part from Commit's own list. Your words, my legs are sore, and today's legs day are the facts. Siri fills Sore: Legs, then Commit's own rules check the plan: upper body today, legs move to next week. Supplied context does not retrain the model, and a persona does not supply factual knowledge or enforce permissions.InstructionsPick from Commit’s listYour wordsMy legsare soreToday: legs daySiri picks Sore: LegsApp checks the planInstructions guide. Facts ground. Code checks.
Instructions guide. Facts ground. Code checks.
Swipe to explore the diagram.
Read the full explanation
No context · generic

My legs are sore.

“Sorry to hear that. Try to rest today.”

The model doesn’t know your plan or what to fill in.

With context · personal

My legs are sore.

Legs day Today’s planLegs · Glutes · Core · Arms · Back Commit’s listSore Field

“Sore: Legs. Commit moves legs to next week: upper body today.”

Same model, with the context it needs.

What you said, today’s plan and Commit’s own list of body parts give Siri useful context. Siri fills one field; Commit’s own rules decide the workout. These response bubbles are illustrations; the recorded screen shows the real result.

Words → a checked plan.

Commit after the Siri check-in: Adapted to your check-in. Sore legs, so Upper body today. Legs and Glutes move to next week.
Recorded Commit screen. After “My legs are sore”, Commit’s own rules swap today’s workout: upper body today, legs move to next week.Recorded · Commit. Sore legs → upper body today.
Implementation and source notes

How to do it

// Siri may only fill a body part from Commit's list.
enum BodyPart: String, AppEnum {
  case legs, glutes, core, arms, back
}
struct LogSoreness: AppIntent {
  @Parameter var part: BodyPart
  func perform() async throws -> some IntentResult {
    try CheckIns.saveToday(sore: part)   // Commit's data
    Plan.adaptToday(for: part)           // Commit's rules
    return .result()
  }
}

The default model runs on the device. Enabled online tools can send requests to external services.

Relate

Like telling a coach how you slept before asking for today’s workout.

Instructions, facts and retrieval

A persona is not a knowledge source. Commit’s App Intent tells Siri what it may fill: one body part from Commit’s own list (Legs, Glutes, Core, Arms, Back). What you said (“my legs are sore”) and today’s plan supply the facts. Siri fills the field; Commit’s own rules then check the plan: upper body today, legs move to next week.

For a factual question, retrieve the relevant source before answering instead of letting the model invent it.

RAG, in a larger app: split documents into useful passages, index them, retrieve relevant passages for the question, then supply those passages and source references to the model. Embeddings and a vector database are one implementation, not a requirement for every app. Retrieved content is context, not permanent training or a guarantee of truth.

Apple: instructions and tools · Retrieval and semantic search

Tools

What can your app connect the AI to?

Ask

Your app can read it, or do it. What would you plug in?

Your turn

It can read

  • Your dataSwiftData · files · your backend
  • Search your dataSpotlight semantic index · your own vector index (FAISS-style)
  • Phone sensorscamera · microphone · location · Health · LiDAR
  • Modelson-device · Private Cloud Compute · Claude / Gemini · your Core ML model
Commit: Siri runs its shortcut

It can do

On the phone

  • Add to Calendar
  • Set a reminder
  • Draft a message
  • Pay, add a pass
  • Open directions
  • Play music
  • Log to Health
  • Run a home scene
  • Siri or a Shortcut
  • Save or share a file
  • Drive a robot
  • Send a notification

Your server and APIs · the request leaves the phone

  • Your own server
  • A weather API
  • Building systems
  • MCP tools

The person confirms anything that sends, pays or moves.

The limits: its own sandbox · the user’s permission · short background time · a small on-device context.

Relate

Anything the app can read becomes context. Anything it can do becomes a tool, or a Siri action.

Foundation Models · App Intents · Core Spotlight · HomeKit · Core Bluetooth · EventKit · PassKit · MusicKit

Why build an app?Why an app, Xcode 27’s agent, your coding partner

Start here · Why an app

Why build an app, not just a website?

Ask

Who has built a website? What did you need besides the code?Who has built a website? What would an app add?

WEB PAGEfar awayThe model is on a serverWords leave the phoneYou pay for each answerCOMMITIt lives on the home screenSiri, camera, HealthData can stay on the phoneA website you visit. An app lives with you.Its model and your data can stay on the phone.
A website you visit. An app lives on the home screen, next to your data.
Swipe to explore the diagram.
Read the full explanation

A website

  • Reach people through a browser
  • Choose local storage or a backend
  • Can support offline features
  • Browser APIs and service integrations

A native Apple app

  • Runs on the user’s own device
  • Direct access to Apple’s AI frameworks
  • Personal data can stay on the phone
  • Native Siri, camera and HealthKit integrations

Both can use a backend, and both need deliberate data choices. This workshop chooses a native app for direct access to the Apple AI and system features being demonstrated.

A websiteAnyone with the link opens it in a tab.
An appOn the home screen, with Siri, camera, Health.
Your dataCan stay on the phone.

Start this weekend

  1. Install Xcode and add a coding agent
  2. Build one screen that fixes your own problem
  3. Keep data on the phone; add a database only if needed
  4. Share it with five friends on TestFlight

Android Studio and Google Play testing offer a comparable build, device-test and distribution path.

Relate

An app is personal: it lives in your pocket, right next to your data.

Start here · Xcode 27

Xcode 27: an agent that knows Swift

Ask

Who has used AI to write code? What went wrong?

An illustrative agent plan sits beside real screenshots of the Passage Swift project in Xcode and the app running in Simulator. The coding agent plans, edits Swift, builds and tests, then opens a preview for you to review. It uses Xcode's tools, can be backed by different agents or a local model, and takes skills and MCP plugins.Xcode 27Plan mode“Show a route preview.”Inspect the view Edit SwiftUI Run the buildPlanCodeBuild + testPreviewYou reviewXcode tools · Your choice of agent · Skills and MCP plugins
Plan, code, build, preview, then you review.
Swipe to explore the diagram.
Read the full explanation
  1. Plan
  2. Code
  3. Build + test
  4. Preview
  5. You review
Plan modeThe agent asks questions and writes a plan before any code.
Xcode’s own toolsBuild, test, previews, the simulator and Apple’s docs.
Any agentAnthropic, OpenAI and Google, or any agent through ACP, even a local MLX model.
PluginsAdd skills and MCP tools.

How to do it

  1. Xcode › Settings › Intelligence: add an agent
  2. Start in plan mode and answer its questions
  3. Read the plan, then let it build, test and preview
  4. Keep a plan file the agent reads first

WWDC26 · 382 Special presentation · 227 UI prototypes with agents

Relate

Like a pair programmer who can also run the build and the tests.

Start here · Coding agents

Pick your coding partner

Ask

Which AI would you trust to write your app’s code? Why?

Your turn
Pick your coding partner. Three run in the cloud and your code goes to them: Claude from Anthropic, Codex from OpenAI and Gemini from Google. One runs on your Mac: your own model, and the code stays on the Mac. Give two of them the same task and compare Swift quality, cost, privacy and working offline.In the cloud · your code goes to themClaudeAnthropicCodexOpenAIGeminiGoogleOn your MacYour modelstays on the MacSame task to two of them. Compare:Swift qualityCostPrivacyOffline
Three in the cloud, one on your Mac.
Swipe to explore the diagram.
Read the full explanation
Anthropic

Claude

Claude Agent, Anthropic’s coding agent in Xcode. The one I used to build the app you’ll meet next.

OpenAI

Codex

OpenAI’s coding agent, also in Xcode.

Google

Gemini

Google’s agent, in Xcode 27.

Local · on your Mac

Your own model

Any agent through ACP, even an MLX model. Code stays on the Mac, but small models are weaker.

Compare them on Swift quality, cost, privacy (cloud agents send your code to the provider) and working offline. These agents write the app. The models your users run come later in the talk.

How to do it

  1. Xcode › Settings › Intelligence: sign in to a provider
  2. Or add a locally hosted model (an MLX server)
  3. Give two agents the same small task, then compare
  4. Check the cost, and what code leaves your Mac

WWDC26 · 382 Special presentation · 232 Local agents with MLX

Relate

Like choosing a pair-programming partner: give two the same task and compare.

Overview

Developer stories

Commit’s data choices and Passage’s working AI features.

Trace the selected data path: the phone holds local records and local inference; enabled network tools or hosting can send chosen information to an outside service. Ask what leaves, who receives it and who keeps it.Your records Your modelWhat leaves?Who receives it? Who keeps it?Trace the data before you choose.
Trace the data before you choose.
Swipe to explore the diagram.

Two real apps, two data choices

Commit · Apple Health read on the phone
Passage · records and model on the device

Commit: TestFlight · Passage: recorded Simulator capture

Live in this room

Scan once and your phone follows the talk. Suggest a small improvement to Passage, choose one to try, and answer live questions as we go.

Answer live questions
Scan to join

Resources

Try three things this week, from the fm command to a Siri action, then read the sessions and open-source projects behind the talk.

See all resources

*Private Cloud Compute has no cloud API cost if your app is in the App Store Small Business Program and has fewer than 2 million first-time downloads (developer.apple.com/machine-learning).

Concepts and recordings

Open full-size captureRecorded simulator screen of Passage