# ProstDev — Full Content Export > ProstDev — where MuleSoft mastery meets fun. Easy-to-follow video tutorials, AI experiments, and deep-dive articles on MuleSoft, DataWeave, Anypoint Code Builder, and integration. --- # Blog Posts ## 3 AIs Design API-Led Connectivity in MuleSoft: Claude vs CurieTech vs MuleSoft Vibes Source: https://prostdev.com/post/ai-showdown-3-ais-design-api-led-connectivity-in-mulesoft | Published: Jul 14, 2026 | Category: Opinion Three AIs. One MuleSoft architect brief. Zero human code. I handed **Claude**, **CurieTech AI**, and **MuleSoft Vibes** the exact same prompt — *design a 3-layer API-led connectivity network for a Customers + Orders domain* — and let each one build the whole thing on its own. Then I scored the results the way I'd review a real submission: contract-first design, business rules, error handling, tests, and version hygiene. This is Round 2 of **AI Showdown: MuleSoft Edition (2026) 🥊**. Round 1 pitted two AIs against DataWeave puzzles; this time the task is a full API-led topology, and there's a third contestant in the ring. Every solution is in the GitHub repo: 🔗 [github.com/alexandramartinez/ai-showdown-api-led](https://github.com/alexandramartinez/ai-showdown-api-led) ## TL;DR All three built the three layers (Experience → Process → System) and implemented **all 11 operations**. The differences were in the engineering discipline underneath: - **Claude** (Opus 4.8, MAX effort) was the **most disciplined build** — contract-first RAML, a parent reactor POM, and the *only* solution whose data survives a restart. Its weak spot: the oldest runtime of the three. - **CurieTech AI** was the **most complete and granular** — five apps split per entity, the most tests (41 MUnit cases), the richest error handling, and a dedicated status endpoint. Its weak spot: version drift across those five apps with no central POM. - **MuleSoft Vibes** (Sonnet) was a **fast, runnable demo** on the freshest connectors — but no API specs, no APIkit, no tests, no error handler, no docs, and writes don't persist. - **None** of the three added a security layer, and **none** used the true latest Mule runtime. ## The brief The prompt asked each AI to *think like a MuleSoft architect* and build an API-led connectivity example with all three layers for a Customers + Orders domain. The ground rules: - **Three layers**: Experience, Process, System. - **Mock and seed test data in the System layer**, as if it were wired to a real database (no actual DB connection yet). - **At least 3 APIs** (one per layer) — split further if it makes architectural sense, but don't build one giant do-everything API and don't repeat code. - **Use the latest** Mule, Maven, and DataWeave versions. And the 11 operations the network had to cover: 1. List all customers 2. Get one customer's details 3. Get a customer's orders 4. Get one order's details 5. Create a customer 6. Edit a customer 7. Delete a customer **only if they have no orders** 8. Create an order attached to a customer 9. Edit an existing order 10. **Cancel** an order (a status change, *not* a delete) 11. Delete an order Two of those are business-rule traps on purpose: #7 (block the delete when orders exist) and #10 (cancel ≠ delete). ## The contestants - 🟦 **Claude** — Opus 4.8, MAX effort - 🟨 **CurieTech AI** — model undisclosed - 🟩 **MuleSoft Vibes** — Sonnet (effort unknown) > [!NOTE] > These aren't necessarily comparable tiers of compute — the models and effort levels differ, and one vendor doesn't disclose its model. Read this as "what each tool produced from the same brief," not a controlled benchmark. ## The scorecard | Dimension | 🟦 Claude | 🟨 CurieTech | 🟩 MuleSoft Vibes | |---|---|---|---| | **Apps** | 4 (1 exp · 1 prc · 2 sys) | 5 (1 exp · 2 prc · 2 sys) | 4 (1 exp · 1 prc · 2 sys) | | **All 3 layers** | Yes | Yes | Yes | | **Parent/reactor POM** | Yes | No | No | | **Contract-first (RAML)** | Yes, all apps | Yes, all apps | No specs | | **APIkit** | Yes | Yes | Hand-rolled HTTP listeners | | **11 operations** | 11/11 functional | 11/11 functional | 11/11 coded, writes don't persist | | **Business rules (#7, #10)** | Both correct | Both correct | Both correct (on read-only data) | | **Security layer** | None | None | None | | **Error handling** | Global handlers, all apps | Global handlers + 502 gateway | None | | **Data persistence** | Across requests **and restarts** | In-memory (lost on restart) | Doesn't persist | | **MUnit tests** | 18 | 41 | 0 | | **Docs / rationale** | Plan + READMEs | READMEs per app | None | | **Version consistency** | Centralized | Drift across apps | Consistent | ## 🟦 Claude: the most disciplined build Claude produced four apps and, crucially, wired them together with a **parent reactor POM** so every app inherits the same dependency versions from one place. That's the single move that separates "four Maven projects in a folder" from "a governed project." It went **contract-first**: RAML 1.0 specs for every app, scaffolded with APIkit, with all 11 operations defined in the spec *and* implemented. Error handling is a global handler in all four apps covering 400/404/409, plus a 500 trait in the spec. The standout: Claude added the **File connector** to the System layer, so seeded data actually **persists across restarts**. It's the only one of the three where you can `POST` a customer, restart the app, `GET` it back, and it's still there. Both business-rule traps are handled — a 409 when you try to delete a customer with orders, and a status flip to `CANCELLED` for cancel-vs-delete. The catch: it shipped on **Mule 4.9.0**, the oldest runtime of the three, with the oldest plugin versions to match. Rock-solid, slightly behind. ## 🟨 CurieTech AI: the most complete and granular CurieTech went the furthest on topology — **five apps**, splitting both the Process and System layers per entity (customers vs. orders). That granularity paid off in coverage: **41 MUnit tests** (more than double Claude's), the richest error handling of the three (including proper **502 gateway** semantics between layers), and the cleanest take on the cancel rule — a dedicated `PATCH /orders/{id}/status` endpoint instead of overloading a `PUT`. It also nailed the seed-data detail that tests the business rules: 8 seed orders across 5 customers, with one customer (`CUST-005`) deliberately left order-free so you can actually exercise the "delete only if no orders" path. The catch: with five independent apps and **no parent POM**, versions drifted. The runtime ranged from 4.9.3 to 4.11.3 across apps, and APIkit/MUnit/HTTP-connector versions were similarly spread. Data lives in **ObjectStore**, so it persists across requests but is lost on restart. Feature-complete and well-tested — just missing the central version governance a parent POM gives you for free. ## 🟩 MuleSoft Vibes: a fast, runnable demo MuleSoft Vibes produced something that runs and uses the **freshest connectors** — it was the only contestant on the current HTTP connector (1.11.3) — with **consistent versions** across its four apps. But it skipped most of what makes API-led *API-led*: **no RAML specs, no APIkit** (hand-rolled HTTP listeners instead), **no MUnit tests, no global error handler, and no docs**. The operations are all coded and return the right status codes, but the data is read-only — a `GET` right after a `POST` returns a 404 for the record you just "created," because nothing actually persists. The business-rule logic is correct on paper; it just runs against data that never changes. Great for a quick "look, it works" demo — not something you'd hand to a team. ## The version reality check The brief said "use the latest." Nobody fully did — and it's a useful reminder that AIs anchor to whatever versions dominated their training data, not to what shipped last week. | Component | 🟦 Claude | 🟨 CurieTech | 🟩 Vibes | Latest GA | |---|---|---|---|---| | Mule runtime | 4.9.0 | 4.9.3–4.11.3 | 4.11.2 | 4.12.0 | | HTTP connector | 1.10.3 | 1.10.4–1.11.1 | 1.11.3 | 1.11.3 | | mule-maven-plugin | 4.3.0 | 4.5.3 | 4.7.0 | 4.10.0 | | APIkit | 1.11.3 | 1.10.4–1.11.12 | — | 1.12.1 | | MUnit | 3.4.0 | 3.6.3 | — | 3.7.1 | | Java | 17 | 17 | 17 | 17 | All three picked **real, released versions** and correctly used **Java 17** — no hallucinated version numbers, which is genuinely reassuring. But none reached **Mule 4.12.0**, and the plugin versions trailed further behind than the runtimes did. If you're using an AI to scaffold a project, pin your versions yourself. ## The gap all three left Every contestant skipped a **security layer** entirely — no client-ID enforcement, no policies, nothing gating the Experience API. The brief didn't explicitly ask for it, but "think like a MuleSoft architect" arguably should have surfaced it. It's the clearest example of AIs doing exactly what's asked and no more: none of them pushed back with "you probably also want auth here." ## So who wins? Depends on what you're optimizing for: - **Want a foundation to build on?** 🟦 **Claude.** The parent POM and real persistence mean you can hand this to a team and extend it without fighting version chaos. - **Want the most coverage and granularity out of the box?** 🟨 **CurieTech AI.** More tests, more error handling, a cleaner status endpoint — just add a parent POM and reconcile versions. - **Want a quick runnable demo to show the shape of API-led?** 🟩 **MuleSoft Vibes.** Fastest to something that runs, as long as you don't need it to remember anything. The meta-takeaway: all three are genuinely useful scaffolding tools, and all three need a human architect to close the same gaps — security, version pinning, and (for two of them) persistence. AI gets you to a working three-layer network fast. It doesn't yet get you to production. ## What's next? **AI Showdown: MuleSoft Edition (2026)** is an ongoing series comparing AI coding tools on the same MuleSoft and DataWeave challenges. Round 1 was DataWeave puzzles; this was API-led architecture. 🎯 What should the next round test — and which AI should join the ring? Drop a comment on YouTube or reply to this post. 🔔 Subscribe so you don't miss the next round: [youtube.com/prostdev](https://www.youtube.com/prostdev) --- ## Migrating a Blog From Wix to Astro: What Breaks Source: https://prostdev.com/post/migrating-a-blog-from-wix-to-astro | Published: Jul 9, 2026 | Category: Guides So I moved this whole blog off Wix. All 138 posts now live on a free static [Astro](https://astro.build/) site hosted on GitHub Pages, no monthly bill, and I own every byte of the content. If you're thinking about the same jump, here's the thing nobody warns you about: a Wix to Astro migration that finishes without a single error can still be quietly missing half your content. I learned that the expensive way. This post walks through what the export drops, why it drops it, and how I got it all back, so you can skip the part where you migrate the same catalog twice. ## TL;DR This is a field report, not a step-by-step tutorial. Wix silently drops a lot on export (code snippets, images, the video, galleries, buttons, comments, and it mangles tables and nested lists), all with no error to warn you. Below is what breaks, why, and how I recovered every piece without fabricating anything. If you just want the tools, the extractor and the full playbook are open-source: **[github.com/alexandramartinez/wix-to-astro](https://github.com/alexandramartinez/wix-to-astro)**. > [!NOTE] > I did this migration with Claude Code as my pair. Everything below is what actually happened on my posts, not a theory. The recovery scripts and the full playbook are in a separate repo, linked at the end. ## What "migrating from Wix to Astro" actually means A Wix to Astro migration is really two jobs, not one. The obvious job is moving your posts into a new format: Wix renders your blog, Astro builds static pages from Markdown (or MDX) files. The hidden job is *recovering the content the move leaves behind*, because Wix keeps a surprising amount of your post outside the HTML a migration script can see. Get the first job right and the site builds. Miss the second and you ship a blog that looks complete and is missing code snippets, images, and videos. That second job is where all my time went, so that's most of what this post is about. ## The first mistake: letting an AI summarize your posts My very first attempt fetched each post through an AI summarizer to turn the page into Markdown. It felt clever. It was a disaster. A summarizer paraphrases your prose and **silently drops every code block**. For a MuleSoft tutorial blog, that's the entire point of the post gone. I didn't catch it until I'd run the whole catalog through, and I had to redo the migration from scratch. Here's the fix, and it's the single most important lesson I can give you: don't summarize, *read the raw server HTML directly*. A small deterministic script that pulls the page and converts the real HTML tags to Markdown is lossless. It keeps your exact words, your links, your fenced code, all of it. Boring beats clever here. > [!WARNING] > A migration that runs without errors is not the same as a migration that's complete. Wix renders a big chunk of your post in the browser, so it was never in the HTML your script downloaded. No error fires because nothing knew that content was supposed to be there. ## What Wix renders client-side (so it was never in the HTML) This is the core insight, and it explains almost every gap I hit. Some of your post lives in the server HTML, and some of it Wix only assembles in the reader's browser. A migration script sees the first kind and is completely blind to the second. Here's what turned out to be client-side, and therefore missing after a clean export: - **Code snippets.** On Wix these were GitHub Gist embeds, little iframes that fill in on load. The code is never in the server HTML, so every snippet vanished. The tell on a live post is a line like "paste this in:" with nothing after it. - **The companion YouTube video.** Wix rendered it as a client-side player widget. Gone, leaving an orphaned "Prefer video? Watch it here" line pointing at nothing. - **Inline images.** These came through as a special Wix image block a first-pass regex never matched. Across my catalog that was 1080 images spread over 114 posts, all missing on the first real pass. - **Image galleries and buttons.** Multi-image carousels and styled call-to-action buttons (like a "Solve on the Playground" link) are the same story: client-rendered, so dropped. - **Reader comments.** Wix loads the comment thread in the browser too, so none of it was in the HTML. Some of those comments were genuinely useful (a reader posting a cleaner DataWeave solution, say), and I didn't want to lose them. None of these threw an error. They just weren't there, and you only notice when you actually read the migrated post. ## What migrated, but came out wrong A second, sneakier category: content that made it across but got mangled on the way, because the HTML-to-Markdown step didn't handle the shape Wix used. - **Tables flattened into loose paragraphs.** A two-column table became alternating one-line paragraphs, so the rows and columns were just gone as structure. The converter had no rule for ``, so each cell leaked out on its own. - **Nested lists collapsed.** Wix nests a sub-list *inside* its parent list item, and a naive match merged the parent with its first child and lost the indentation. - **Bold text glued to neighbors.** Wix likes to tuck the spacing inside the emphasis, so you'd get `**Pick this **option` with the bold swallowing the space, or one bold word split into two runs. - **Doubled quote marks on blockquotes.** The typography styling adds its own curly quotes, and the migrated text already had the author's quotes, so quotes showed up twice. These are all fixable, but you have to *know to look*. A build passes fine with a flattened table in it. ## Recovering it all without fabricating anything Here's the rule I held to the whole way through, and I'd push you to do the same: **never invent the missing content.** Every recovered code snippet came verbatim from the live rendered page or from my own source files, never from guessing what the code "probably" was. When a match was fuzzy (which dropped snippet belongs in which gap), I confirmed it by hand instead of letting a script auto-place it. That discipline is what makes the migrated post trustworthy. A tutorial with a plausible-but-wrong code block is worse than one with an honest "code was here" gap, because the reader can't tell it's wrong. So when in doubt, I left a visible hole and came back to it rather than filling it with a fabrication. The reader comments were the purest version of this rule. Unlike code or images, there's nothing to auto-recover, no script can bring them back, so I copied the handful of genuinely-useful ones straight off the live page, word for word, and preserved them as attributed notes under the post. The rest I let go. No paraphrasing, no inventing a helpful-sounding reply that nobody actually wrote. ## The upgrade the move unlocked The migration wasn't just damage control. Moving to my own Astro site let me fix things Wix never let me touch. The big one was accessibility. When I pulled the inline images across, 93% of them had no alt text at all, 977 empty out of 1047. That's a real problem for screen-reader users and for image search. So I went through and wrote genuine alt text by actually looking at each image, not keyword-stuffing, just describing what's in the shot. The other win is for AI. Every post on the new site has a plain-text `.md` version and feeds an `llms.txt`, so an answer engine can read the real content instead of scraping a JavaScript-heavy page. My old Wix blog was basically invisible to that whole world. Now the content is built to be quoted. ## Should you do it? First, to be fair to Wix: it's genuinely good at a lot of things. If you're running a shop, selling services or subscription plans, taking bookings, managing events, or you want a full CMS with a visual editor and no code, Wix handles all of that out of the box, and handles it well. The payment, scheduling, and store tooling alone can be worth the monthly fee. But if you're *just* hosting a website, mostly writing, and not accepting payments or running any of that commerce machinery, then you're paying every month for a lot of features you don't use, and getting a slower, less portable site than you could have for free. That's exactly the case where I'd make the move. So: if you're paying Wix monthly and your content is mostly writing, yes, I think it's worth it. You get free hosting, a faster site, full ownership, and content that both search engines and AI can actually read. Just go in with the right expectation: budget real time for the *recovery* phase, not only the export. The export is the easy 20%. Getting your code, images, videos, and tables back, verbatim and complete, is the other 80%, and it's the part that decides whether readers trust the result. ## What's in the repo I packaged everything I trusted into a separate open-source repo so you don't have to rediscover any of this. In short, it has: - **The extractor.** A small, dependency-free Node script that reads a Wix post's raw server HTML and converts it to clean Markdown verbatim, pulls the full-res original images, and reads the title, author, and dates from the page. There's a one-post command and a batch driver that migrates a whole blog. - **A QA linter.** After extraction it scans the output and prints exactly which posts are missing something (an empty image alt, a "paste this in:" line with no code after it, a flattened table, an MDX build trap), so you're not hunting blind for the gaps. - **The playbook.** A plain-English rundown of every trap above (the summarizer mistake, each thing Wix renders client-side, the tables and lists that come out wrong, the Astro/MDX build gotchas) with how to spot and fix each one. There's a version written for an AI pair too, so if you're migrating with Claude Code or Cursor it follows the same never-fabricate discipline. It does the honest, lossless part, tells you where the gaps are, and is explicit about what no script can recover, so you know exactly where the by-hand work starts. > [!BUTTON] > [Go to the GitHub repo](https://github.com/alexandramartinez/wix-to-astro) If you want the engineering underneath it (the tokenizer design, the Wix-specific paper cuts, and a zero-width regex that quietly pegged a CPU core), I wrote a deeper technical version on [dev.to](https://dev.to/devalexmartinez/migrating-a-wix-blog-to-markdown-the-server-html-trap-and-the-zero-width-regex-that-pegged-my-cpu-1lhc). If you hit a Wix quirk I didn't cover, I'd genuinely like to hear about it. Come find me on [YouTube](https://www.youtube.com/@prostdev) or wherever you found this post, and happy migrating. :-) --- ## Keep Your .claude Folder Private with a Git Submodule Source: https://prostdev.com/post/private-claude-folder-git-submodule | Published: Jul 5, 2026 | Category: Tutorials I build in the open. This site's whole [codebase](https://github.com/ProstDev/prostdev.github.io) is a public GitHub repo, and I like it that way. But my `.claude/` folder is a different story. It holds the skills, subagents, and `CLAUDE.md` instructions I've spent months tuning, plus notes I'd rather not hand to a scraper. I wanted the project public and that one folder private. The fix is a [git submodule](https://git-scm.com/book/en/v2/Git-Tools-Submodules). `.claude/` becomes its own private repo, wired into the public project. That folder holds all my [AI agent skills](/post/what-are-ai-agent-skills-and-how-to-use-them), subagents, and instructions, so keeping it private matters. There's one snag with `CLAUDE.md` that I'll walk you through, because Claude Code expects it at the repo root but I want the real file inside the private folder. A relative [symlink](https://en.wikipedia.org/wiki/Symbolic_link) solves it. This is exactly how prostdev.com is set up. Here's the whole thing, start to finish. > [!TIP] > You can follow these steps by hand, or you can hand them to your AI. Every step below is a plain command with the reasoning spelled out, so a coding agent like Claude Code can run the whole thing for you. If you go that route, paste the steps in, tell it your GitHub owner and the repo names you want, and ask it to stop and check with you before it creates the private repo or force-pushes anything. Those are the two steps you don't want an agent doing unsupervised. > [!NOTE] > This guide is written for macOS and Linux. The `CLAUDE.md` symlink step relies on POSIX symlinks, which Windows checkouts don't always materialize the same way. ## What you'll end up with Two repos instead of one: - **A public repo.** Your actual project (this site, in my case). Anyone can read it. - **A private repo.** Just the contents of `.claude/`: your skills, agents, docs, and the real `CLAUDE.md`. The private one is pulled into the public one as a submodule at the `.claude/` path. To anyone browsing your public repo, `.claude` looks like a pointer to a commit in another repo. They can't see what's inside unless you grant them access. ![GitHub's file view of the public prostdev.github.io repo. The file list shows a .claude entry, and the CLAUDE.md file is open, labeled "Symbolic link · 1 lines (1 loc) · 17 Bytes" with its entire contents being the single line ".claude/CLAUDE.md".](../../assets/blog/private-claude-folder-git-submodule-1.png) Two extra payoffs beyond privacy. First, you can **reuse that internal config across projects.** Point a second project's submodule at the same private repo and every skill comes along. Second, your `.claude/settings.json` (which can hold machine-specific paths) stays out of both repos entirely, which I'll cover near the end. ## Before you start You'll need the [GitHub CLI](https://cli.github.com/) (`gh`) authenticated, and a project that already has a `.claude/` folder with real content in it. Confirm `gh` is signed in: ```bash gh auth status ``` If that prints your account, you're good. Everything below runs from the root of your public project. > [!WARNING] > A few of these steps rewrite git history and delete the local `.claude/` folder before re-adding it. Commit or stash any uncommitted work first, and don't run this on a folder whose contents you haven't pushed somewhere safe. ## Step 1: Move the real CLAUDE.md inside .claude Claude Code reads `CLAUDE.md` from your project root. But I want the *real* file versioned in the private repo, so it needs to live inside `.claude/`. We'll reconcile that with a symlink in a later step. For now, move it in: ```bash git mv CLAUDE.md .claude/CLAUDE.md ``` If your `CLAUDE.md` wasn't tracked yet, use plain `mv` instead. If you already keep it at `.claude/CLAUDE.md`, skip this step. Either way, the source of truth is now `.claude/CLAUDE.md`. ## Step 2: Ignore your settings files in both repos `settings.json` and `settings.local.json` hold per-machine and personal config (local paths, permission choices). They shouldn't live in *either* repo. Add them to the `.gitignore` inside `.claude/`: ```gitignore title=".claude/.gitignore" # Claude Code: ignore personal/local settings (keep shared skills/agents/CLAUDE.md) settings.json settings.local.json .DS_Store ``` Then make sure your project root's `.gitignore` ignores them too, so they can't sneak in from the top: ```gitignore title=".gitignore" settings.json settings.local.json ``` If either file is *already* tracked, untrack it without deleting your copy on disk: ```bash git rm --cached .claude/settings.json .claude/settings.local.json ``` The `--cached` flag is the important part. It removes the file from git's index but leaves the actual file sitting in your folder, working as before. ## Step 3: Create the private repo from your .claude contents Now we give `.claude/` its own git history and push it to a new private repo. To avoid disturbing the main project's git state, do this in a throwaway copy of the folder, *outside* your project tree: ```bash cp -R .claude /tmp/claude-internal cd /tmp/claude-internal rm -rf .git # drop any leftover git metadata from the copy git init git add -A git commit -m "Initial import of .claude internal config" ``` Add a short `README.md` first if the folder doesn't have one, so the repo isn't bare. Then create the private repo and push, using `gh`: ```bash gh repo create your-name/project-internal --private --source=. --remote=origin --push ``` Swap `your-name/project-internal` for the owner and name you want. `--private` is what keeps it locked down, `--source=.` points it at this folder, and `--push` sends the commit up immediately. > [!IMPORTANT] > This is the one step that publishes something. Double-check the repo name and the `--private` flag before you run it. Everything before this was local. ## Step 4: Replace the local folder with a submodule Back in your public project, remove the plain `.claude/` folder from git and re-add it as a submodule pointing at the repo you just created: ```bash cd /path/to/your/public/project git rm -r --cached .claude rm -rf .claude git submodule add https://github.com/your-name/project-internal.git .claude ``` The `git rm -r --cached` stops tracking the folder's files directly. The `rm -rf` clears the local copy (it's safe now, everything is up in the private repo). And `git submodule add` clones the private repo into `.claude/` and records the link. You'll see a new `.gitmodules` file. It should look like this: ```ini title=".gitmodules" [submodule ".claude"] path = .claude url = https://github.com/your-name/project-internal.git ``` Confirm the submodule is healthy: ```bash git submodule status ``` A line beginning with a space and a commit hash, ending in `(heads/main)`, means the submodule is checked out and clean. ## Step 5: Recreate CLAUDE.md as a symlink Here's the trick that ties it together. The real file is now at `.claude/CLAUDE.md`, but Claude Code looks for `CLAUDE.md` at the project root. So we create a symlink at the root that points into the submodule: ```bash ln -s .claude/CLAUDE.md CLAUDE.md ``` Use the **relative** target `.claude/CLAUDE.md`, not an absolute path. A relative link resolves no matter where the repo is cloned; an absolute one would break on anyone else's machine. Verify it: ```bash ls -la CLAUDE.md ``` You should see something like `CLAUDE.md -> .claude/CLAUDE.md`, and the file will be tiny (about 17 bytes, the length of the target path). That's the tell that it's a symlink and not a copy. Reading it should show your real instructions: ```bash cat CLAUDE.md # prints the content of .claude/CLAUDE.md through the link ``` Now stage it. Git stores a symlink as a special entry (mode `120000`), not as a duplicate of the file's contents: ```bash git add CLAUDE.md ``` When you commit, `git ls-files -s CLAUDE.md` will confirm the `120000` mode. That's how you know git recorded the link itself rather than inlining the file. ## Step 6: Learn the two-repo commit rule From here on, your project is **two git repos**, and where a file lives decides which one you commit to. - Edit a **project file** (source code, a config, a blog post)? That's one ordinary commit in the public repo. - Edit an **internal file** (a skill, a subagent, a doc, or `CLAUDE.md`)? That's a **two-step commit**. The internal edit goes *inside the submodule first*, then the public repo records the new pointer: ```bash # 1. Commit inside the submodule git -C .claude add CLAUDE.md git -C .claude commit -m "Tweak instructions" git -C .claude push # 2. Bump the gitlink pointer in the public repo git add .claude git commit -m "Bump .claude submodule pointer" ``` The submodule commit has to exist and be pushed *before* the public repo points at it, or you'll reference a commit nobody else can fetch. When you run `git status` in the public repo and see `modified: .claude (new commits)`, that's not a stray change. It's git telling you the pointer is behind and you still owe step 2. For example, this is how the Source Control tab looks like in my local once I've modified files in the `.claude/` folder. First, I have to commit the files at the bottom (`.claude`), and then the files at the top (`prostdev`). Note how the `.claude` change at the top has an `S` for Submodule. ![VS Code Source Control panel showing two separate repositories stacked with their own commit boxes: the prostdev public repo on top with 3 changes, and the nested .claude submodule below with 2 changes including a modified CLAUDE.md](../../assets/blog/private-claude-folder-git-submodule-2.png) ## Step 7: Fix the clone gotcha There's one catch that trips up everyone, including future you on a new laptop. A normal `git clone` of your public repo brings down an **empty** `.claude/` folder. The submodule contents don't come automatically. Clone with the submodule in one shot: ```bash git clone --recurse-submodules https://github.com/your-name/your-project.git ``` Already cloned the plain way? Pull the submodule in after the fact: ```bash git submodule update --init ``` Because this bites everyone, say so in your public repo's README. A single line ("clone with `--recurse-submodules`") saves the next person a confused ten minutes wondering why Claude Code has no skills. ## Purge the private files from your git history Step 4's `git rm -r --cached .claude` stops git from tracking those files *going forward*. It does not remove them from the commits you already made. Git history is append-only by default, so every earlier commit still holds the full contents of `.claude/`, viewable by anyone who checks out an old commit. If the whole point is privacy, "untracked from now on" is not enough. The ideal is to never let the internal files reach the public repo's history in the first place: do this split before your first commit, or at least before you push. When that ship has already sailed, here's what to do based on where you are. > [!IMPORTANT] > Removing a file from the latest commit is not the same as removing it from history. Until you rewrite history, `git log` and any old checkout still expose it. ### If they never got committed Best case. If `.claude/` was gitignored or split out before your very first commit, there's nothing in history to clean. Confirm it: ```bash git log --all --oneline -- .claude ``` No output means the path never appeared in a commit, and you're done. ### If they're only local and not pushed yet This is the easy one, because rewriting history is safe when nobody else has your commits. The bulletproof move is to collapse everything into one fresh commit of the current, correct state (symlink plus submodule, zero private files). You lose your old commit *messages*, but you get a guaranteed-clean history: ```bash git checkout --orphan clean-history # a new branch with no parent git add -A git commit -m "Public project (private .claude lives in a submodule)" git branch -D main # drop the old history git branch -m clean-history main # promote the clean branch to main ``` Because you started from an orphan branch, the new `main` has exactly one commit and no trace of the old `.claude/` files. You never pushed, so there's nothing to force and nobody to coordinate with. ### If they were already pushed Now the files are public, and two things are true. First, you have to rewrite history *and* force-push it. Second, and this matters more: **treat anything sensitive in those files as already leaked.** Clones, forks, and GitHub's own caches may still hold the old commits, so rotate any API keys, tokens, or secrets that were exposed. A history rewrite is not a substitute for rotation. To rewrite while keeping the rest of your history, use [git-filter-repo](https://github.com/newren/git-filter-repo) (install it once with `brew install git-filter-repo`): ```bash git filter-repo --path .claude --invert-paths ``` That strips every `.claude/` entry from every commit. It also rewrites all your commit hashes and drops the `origin` remote as a safety check, so re-add your remote, re-wire the submodule (repeat Step 4), then force-push: ```bash git remote add origin https://github.com/your-name/your-project.git git push --force origin main ``` Don't want to keep the old history at all? The orphan-branch reset from the previous section works here too. Just add `git push --force origin main` at the end, and warn anyone else who cloned the repo, because their history no longer matches yours. > [!CAUTION] > A force-push rewrites shared history. If other people have cloned or forked the repo, coordinate with them first. And whatever you do, rotate any secret that was ever pushed. Removing it from history does not un-leak it. ## Wrapping up That's the whole pattern: a private submodule at `.claude/`, a relative `CLAUDE.md` symlink so Claude Code still finds its instructions, settings files ignored in both repos, a two-step commit for anything internal, and a clean history with none of those private files ever committed to the public repo. The upfront cost is one afternoon of git wrangling. What you get back is a project you can develop fully in the open without publishing the config you've invested real time in, and an internal toolkit you can drop into the next project by pointing a new submodule at the same private repo. If you set this up and Claude Code suddenly can't see your skills, the first thing to check is whether the submodule actually populated. Run `git submodule status` and `ls .claude/` before you debug anything fancier. --- ## AI Agent Skills: What They Are and How to Use Them Source: https://prostdev.com/post/what-are-ai-agent-skills-and-how-to-use-them | Published: Jul 4, 2026 | Category: Guides AI agent skills are pretty much pieces of information that you save in Markdown files. You can write whatever you want in them, or you can also let your own AI write them for you (or for itself) whenever some new information is needed. There are some standards that are good to follow on how to structure these files to keep your AI's context lower, and hopefully spend fewer tokens! But we'll talk about that in a bit. ## First of all, what really are skills? So as I said before, they're just Markdown files inside folders with specific names on them. :-) The AI would either know which one to call, or you can also manually call it at any point in your session. > [!NOTE] > I'll be talking about my experience with Claude Code because that's what I normally use on my day to day, but skills are reusable in any other AI agent. So what is inside a `SKILL.md` file? It has two parts: 1. A short YAML header (called *frontmatter*) that contains a `name` and a `description`. 2. The instructions themselves, written in Markdown. The name has to match exactly the name of the folder that contains the skill for the AI to run it. The description has to be as useful and descriptive as possible so the AI knows when exactly to run this skill. Try to always answer what this skill does and when to use it, so the AI also knows when it's useful to use or not. This first part is important because when the AI is looking for skills, it either reads only the folder names to filter out the ones that don't fit, or goes and reads the frontmatter of all the available skills to figure out which one is needed and which one is not for the specific job it's trying to accomplish. Let's see an example. This is a simple skill: ```markdown title="SKILL.md" --- name: commit-message description: >- Write a git commit message from the staged diff. Use when the user asks to commit, write a commit message, or summarize staged changes. --- # Write a commit message Read the staged diff, then write a commit message that: - Uses a concise, imperative subject line under 50 characters. - Leaves a blank line, then a body explaining *why*, not *what*. - References an issue number when the branch name contains one. ``` As you can see, the description says what the skill does (writes a git commit message from the staged diff) and when to use it (use when the user asks to commit, write a commit message, or summarize staged changes). After that is the actual skill's instructions to the AI on how you want it to write the commit message exactly. ### Why the name and description matter so much Imagine if the skill was called `commit` and the description was just `write a commit`. Remember that the AI first reads only the frontmatter and not the full body of the skill, otherwise it would defeat the purpose of getting your context under control. So if it just sees "write a commit", it's not really sure what to do with that! Are you asking it to commit your changes? Are you asking it to write some code and commit? Do you want to review the files before the commit? It doesn't even specify that you just want it to write the *message* of the commit. Of course this skill would perform poorly on any AI. This is why the name and the description are so important. Otherwise your AI will have to end up reading all your skills (or none!) to try to figure out what you want, which will cost you more turns, more context, and more tokens. ## Setting up your first skill in Claude Code Claude Code looks for skills in two places, and it loads both automatically when it starts up. The only real difference between them is *scope*: | Where | Path | Use it for | | --- | --- | --- | | **Personal** | `~/.claude/skills//SKILL.md` | Skills you want in *every* project. Your own habits and preferences. | | **Project** | `.claude/skills//SKILL.md` | Skills that only make sense in *this* codebase. Its conventions, its build steps. | Here's how you make your first personal skill: 1. **Make the folder.** The folder name becomes the skill's name, so keep it short and use kebab-case (words joined with dashes): ```bash mkdir -p ~/.claude/skills/commit-message ``` 2. **Create the `SKILL.md` inside it.** Open `~/.claude/skills/commit-message/SKILL.md` and paste in the example from above, or your own. 3. **Keep the `name` and the folder in sync.** Remember, the `name` in the frontmatter has to match the folder name exactly, or the AI won't run it. 4. **Write a description that says *when*.** "Use when the user asks to commit…" is way more useful to the AI than "a commit tool". The clearer the trigger, the more reliably the skill fires at the right moment (and only the right moment). 5. **Restart Claude Code**, or start a new session. Skills get indexed when it starts up, so a brand-new skill gets picked up on the next launch. And that's the whole setup. There's no build step, no config file, no registering anything. The folder *is* the installation. ### Calling a skill Once a skill is installed, there are two ways to trigger it: - **Let the AI decide.** Just describe your task like you normally would. If it matches the skill's description, the AI loads it and follows it on its own. - **Call it yourself by name.** Type `/commit-message` (a slash plus the skill's name) when you want to make sure that specific skill runs. ## Using skills in other AIs Like I mentioned at the top, a `SKILL.md` is just Markdown with a small header, so there's nothing Claude-Code-specific about the file itself. That gives you two ways to use it anywhere else: - **If the tool supports the format**, point it at your skills folder and it loads them the same way: reading the names and descriptions first, and the full body only when it needs it. The [Claude Agent SDK](https://docs.claude.com/en/api/agent-sdk/overview) and the Claude desktop and web apps all understand skills, and more agent tools keep adopting the same `SKILL.md` convention. - **If the tool doesn't support skills natively**, you can still use the content. Just paste the body of the `SKILL.md` right into the conversation as your instructions. You lose the automatic matching, but you keep the workflow you wrote, which is most of the value anyway. So writing your workflow as a skill is never wasted effort, even for an AI that has never heard the word "skill". It's still a really good, reusable prompt. ## Browsing the ProstDev Skills page The [Skills page](/skills) here on ProstDev is a shared library of the real skills I use with Claude Code. Each card is one skill. Here's how to get the most out of it: - **Read the description first**, exactly like the AI does. It tells you *when* the skill is meant to fire, which is the fastest way to know if it fits your workflow. - **Filter by tag** with the chips at the top (`Claude Code`, `MDX`, `AEO`, and so on) to narrow the grid down to a topic. - **Open a skill** to read my own explanation of it, plus the complete `SKILL.md` in a code block you can copy. Grab it, drop it into `~/.claude/skills//`, and tweak the parts that are specific to your own setup. Every skill page is also machine-readable. Add `.md` to the end of any skill URL, like [`/skill/save-session.md`](/skill/save-session.md), and you get the raw Markdown with none of the page around it. That plain version is exactly what you'd hand to an AI. ## Point your AI at llms.txt and let it pick the skill Here's where it gets really handy. You don't have to read the whole library yourself and decide which skill an AI needs. You can let the AI do that, the same way it already picks between its *own* skills. The site publishes an [llms.txt](/llms.txt) file. This is a small, growing web standard (you can read about it at [llmstxt.org](https://llmstxt.org)): a single Markdown index of a site, full of links, written specifically for AI crawlers. ProstDev's `llms.txt` has a section that looks like this: ```markdown title="llms.txt (excerpt)" ## Claude Code skills Reusable Claude Code / AI-agent skills Alex Martinez uses and shares. Each has a machine-readable Markdown version (append `.md` to its URL). - [save-session](https://www.prostdev.com/skill/save-session): End-of-session routine… - [add-faqs](https://www.prostdev.com/skill/add-faqs): Add an FAQ section to a post… - [blog-seo-standards](https://www.prostdev.com/skill/blog-seo-standards): SEO + AEO standards… ``` Notice this is the exact same shape as a skill's own frontmatter: a name and a one-line description of when to use it. That's not a coincidence, it's the same idea one level up. So the flow is: 1. **Give a browsing-capable AI the URL** `https://www.prostdev.com/llms.txt` along with your task. For example, "I'm about to wrap up a coding session, is there a skill for that?" 2. **The AI scans the list of names and descriptions** and matches your task to a skill, just like Claude Code matches your request to a local skill. 3. **It grabs the full instructions** by adding `.md` to that skill's URL (`/skill/save-session.md`) and follows them. You've basically handed the AI a menu it can read and order from by itself. And because it's reading descriptions first, it only pulls the full text of the one skill it actually needs, which (you guessed it) keeps your context lower and your tokens down. ## A few tips to get more out of skills - **Spend your effort on the description.** It's both the trigger and the summary the AI scans. Phrase it as "use this when…". A vague description means the skill never fires when you want it to. - **Keep each skill small and focused.** One job per skill. A skill that tries to do everything matches nothing cleanly. - **Let skills call other skills.** You can chain small skills into a bigger workflow instead of writing one giant file. One "orchestrator" skill can call several focused ones in order, so a big job stays easy to maintain. - **Check them into your repo.** A project skill in `.claude/skills/` travels with the code, so your whole team and every future session get the same behavior for free. - **Refresh them when they go stale.** Skills drift as your tools change. Updating one is usually a quick edit to the body, not a rewrite. > [!TIP] > Not sure your `SKILL.md` follows the conventions? I keep a skill for exactly that: [`claude-md-and-folder-standards`](/skill/claude-md-and-folder-standards). Point Claude Code at it and ask it to check a skill you just wrote, and it'll verify the name and description, the folder layout, the size, and the rest against the standards, so your config stays lean. ## Wrapping up Skills turn your best prompts into permanent, reusable pieces of information your AI can reach for on its own. A skill is just a folder with a `SKILL.md` inside: a `name`, a "use this when…" `description`, and some Markdown instructions. Drop it in `~/.claude/skills/` for Claude Code, paste it into any other AI that doesn't read the format natively, or publish it so other people can copy it. And because a skill's frontmatter and an `llms.txt` index both describe *when* to use each skill in one line, you can hand an AI an `llms.txt` and let it browse and pick the right skill by itself. Start with one skill for the thing you re-type the most. Once you see the AI reach for it without you asking, you'll want to write ten more! Need a head start? Browse the real skills I use with Claude Code, copy the `SKILL.md`, and make it yours. > [!BUTTON] > [Explore the ProstDev Skills library](/skills) --- ## Claude Code vs CurieTech AI: Which Writes Better DataWeave? Source: https://prostdev.com/post/claude-code-vs-curietech-dataweave | Published: May 28, 2026 | Category: Opinion Two AIs. Four DataWeave puzzles. One question: **Which one writes better MuleSoft code?** I put [Claude Code](https://claude.com/claude-code) (running in MAX effort mode) head-to-head against [CurieTech AI](https://platform.curietech.ai) to solve [Advent of Code 2025](https://adventofcode.com/2025) — Days 1 and 2, both parts. Same input. Same expected output. Side-by-side execution times in the [DataWeave Playground](https://dataweave.mulesoft.com/learn/playground). This is also the first video in a new series I'm starting: **AI Showdown: MuleSoft Edition (2026) 🥊** ## TL;DR Both AIs got the **correct answer on all 4 puzzles**. Execution times in the Playground were within milliseconds of each other. The real differences showed up in how they got there — code readability, type usage, error handling, and whether the AI actually runs the code before handing it back. ## The setup - **Claude Code** — VS Code extension, MAX effort mode, Sonnet 4.5 - **CurieTech AI** — tested two ways: - Day 1: via the CurieTech MCP server inside Claude Code - Day 2: directly through the chat at [platform.curietech.ai](https://platform.curietech.ai/) - **Challenges**: Advent of Code 2025, Days 1 and 2 (both parts) - **Benchmarking**: dw::util::Timer::time() in the DataWeave Playground ## What surprised me ### ✅ Execution times were ridiculously close We're talking millisecond-level differences — even running in the browser-based Playground. Sometimes Claude edged out, sometimes CurieTech did. Honestly? Coin-flip territory. ### ✅ CurieTech actually runs the code This is the big one. CurieTech executes the DataWeave it generates before returning the answer. Claude doesn't — it just produces code and hopes for the best. The result: I caught a runtime error in Claude's Day 2 solution that only surfaced when I ran it locally. Had to feed the error back into Claude for a second pass. CurieTech never had that problem. ### ✅ CurieTech writes typed function signatures Look at the difference: Claude (untyped): ``` fun claude(input) = ... ``` CurieTech (typed): ``` fun curietech(input: Array): Number = ... ``` When you're reading code cold (or coming back to it months later), those types make a huge difference. ### ✅ CurieTech adds performance notes On Day 2 Part 1, CurieTech proactively warned me: > "Ranges like this are tiny (~7 IDs), so brute forcing works here. If you ever face huge ranges, you'd want to generate candidates, loop over half-lengths, and concatenate instead of scanning every number." That kind of contextual insight is genuinely useful — especially for folks newer to DataWeave. ### ✅ Claude Code wins on workflow If you live in VS Code or your terminal, Claude Code's integration is hard to beat. File context, multi-step tasks, the whole ecosystem. CurieTech's MCP server brings some of that into Claude Code, but the native experience is still tighter. ### ❌ Claude can ship broken code As mentioned — Day 2 had a runtime error that wasted time. Not a dealbreaker, but worth knowing. ### ❌ CurieTech needed a nudge on Day 1 Part 2 The first attempt got the wrong answer. After I gave it more context about the input format, it produced a verified solution. Slight friction, but the recovery was clean. ## So who wins? Honestly? There's no clear winner. Each has real strengths: - **Pick Claude Code** if you value workflow integration, live in VS Code, and don't mind catching the occasional runtime error. - **Pick CurieTech AI** if you value verified-running code, typed function signatures, and inline performance notes. For someone newer to DataWeave or AI coding tools, **CurieTech is probably the gentler entry point** — the chat UI is approachable and the verified-solution guarantee is reassuring. For experienced devs already deep in the Claude Code ecosystem, Claude Code with the CurieTech MCP server is a great hybrid — you get the workflow you want plus the verification when you need it. ## All the code Every solution, every benchmark script, every Playground link is in the GitHub repo: 🔗 [github.com/alexandramartinez/adventofcode-2025](http://github.com/alexandramartinez/adventofcode-2025) ## What's next? This is the first video in **AI Showdown: MuleSoft Edition (2026)** — an ongoing series comparing AI coding tools on the same MuleSoft and DataWeave challenges. As I work through more Advent of Code days (and test more AIs), I'll keep posting the results here. 🎯 What AI tool should I compare next? Drop a comment on the YouTube video or reply to this post. 🔔 Subscribe so you don't miss the next round: [youtube.com/prostdev](https://www.youtube.com/prostdev) *Looking for the previous series? Check out "Adventures in MuleSoft + AI" on the channel for last year's explorations.* --- ## How I Use ChatGPT to Brainstorm Newsletter Topics Source: https://prostdev.com/post/how-i-use-chatgpt-to-brainstorm-newsletter-topics | Published: Sep 25, 2025 | Category: Opinion Every week, I need a juicy, relevant topic for [AI Picked, Alex Approved](https://www.linkedin.com/newsletters/%F0%9F%94%AE-ai-picked-alex-approved-7348737640038350849/). Lately, I’ve been experimenting with ChatGPT as my brainstorming partner — and it’s been surprisingly effective. Here’s how the process went for my latest edition: 1️⃣ I fed it my previous newsletter topics so it knew what I’d already covered. 2️⃣ I asked for fresh angles, from AI in IDEs to Docs vs AI debates. 3️⃣ We drilled down into real-world developer opinions, research, and stats. 4️⃣ I used it to structure a fast-read newsletter with **fast facts, real quotes, and a deeper dive** for readers who want more. The result: a newsletter draft ready to go, packed with credible sources and actual dev stories — all while keeping my voice front and center. [Read the full edition here](https://www.linkedin.com/pulse/ide-cage-match-can-you-trust-ai-your-editor-alex-martinez--pdvic/). Pro tip: AI is amazing for **idea generation and research**, but the human touch is still essential for context, nuance, and personality. Note that this conversation took around 30 minutes but it did give me two different posts based on the same conversation (the newsletter and THIS post you're reading right now!). ![ChatGPT prompt feeding it the AI Picked, Alex Approved newsletter, then suggesting AI, MuleSoft, and content topics](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-1.png) ![ChatGPT given the list of past newsletter topics, then proposing fresh angles grouped by theme](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-2.png) ![ChatGPT comparing "Docs vs AI" and "IDE Cage Match" topics with a pros and cons breakdown](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-3.png) ![ChatGPT weighing pros and cons of the IDE Cage Match topic and recommending it as the next edition](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-4.png) ![ChatGPT's first newsletter draft comparing expectations vs reality of AI coding copilots](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-5.png) ![ChatGPT listing what devs praise and what breaks with AI IDE tools, with cited sources](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-6.png) ![ChatGPT suggesting hands-on experiments to try and real developer quotes to include](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-7.png) ![ChatGPT framing the editorial angle and offering a TL;DR shareable soundbite for the piece](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-8.png) ![ChatGPT condensing the draft into Fast Facts, What This Means, and My Take sections](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-9.png) ![ChatGPT's "IDE Cage Match" draft with numbered fast facts, takeaways, and a deeper dive section](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-10.png) ![ChatGPT offering real developer testimonials and a "Deeper Dive + Real-Dev Stories" section](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-11.png) ![ChatGPT quoting developer complaints about AI copilots plus supporting research and stats](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-12.png) ![ChatGPT's polished Deeper Dive split into "The Good" and "The Bad" with quoted dev reviews](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-13.png) ![ChatGPT's "The Ugly" and "The Takeaway" sections with a Cursor refactor quote and security stats](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-14.png) ![ChatGPT's near-final newsletter draft combining fast facts, takeaways, and quoted dev reviews](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-15.png) ![Continuation of the draft ending with The Ugly and The Takeaway, then offering a cleaned-up version](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-16.png) ![Cleaned-up "IDE Cage Match" draft with formatted fast facts, what-this-means, and deeper-dive sections](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-17.png) ![End of the draft, then ChatGPT offering to write a short LinkedIn caption to promote the edition](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-18.png) ![ChatGPT drafting a LinkedIn caption, then a separate post about the brainstorming process itself](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-19.png) ![ChatGPT producing a LinkedIn post titled "How I Use ChatGPT to Write About AI in IDEs Without Losing My Voice"](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-20.png) ![ChatGPT's final post styled after this article, recapping the ChatGPT brainstorming workflow](../../assets/blog/how-i-use-chatgpt-to-brainstorm-newsletter-topics-21.png) --- ## How I Use ChatGPT for Writing Without Losing My Voice Source: https://prostdev.com/post/how-i-use-chatgpt-to-create-articles | Published: Sep 18, 2025 | Category: Opinion A lot of people either misuse AI tools or don’t use them at all—even though these tools could make their work so much more efficient. From what I’ve seen, the main reason is simple: most people aren’t sure how to get the best results from them. For me, AI has become part of my writing process. I run a LinkedIn newsletter, and every edition is built with the help of tools like ChatGPT. But here’s the key—I don’t just copy-paste whatever the AI spits out. I guide it with my own tone, I curate the content, and I make sure it’s factually accurate. In other words, the AI helps me move faster, but I stay in control of the message. Take my [latest newsletter on CurieTech AI](https://www.linkedin.com/pulse/curietech-ais-glow-up-chat-first-create-later-alex-martinez--l8ikc) as an example. I used ChatGPT to brainstorm and structure the post, then shaped it into something that sounded like me. That way, I saved time without losing my voice or letting the AI drift off track. Note that this whole conversation took place in less than 10 minutes. ![ChatGPT prompt asking for a newsletter article about the new CurieTech AI chat-first experience.](../../assets/blog/how-i-use-chatgpt-to-create-articles-1.png) ![ChatGPT's first draft titled "A Smarter Way to Work with AI" with a "Why This Matters" bullet list.](../../assets/blog/how-i-use-chatgpt-to-create-articles-2.png) ![Continuation of the first draft with "Example in Action" and "My Take" sections.](../../assets/blog/how-i-use-chatgpt-to-create-articles-3.png) ![Follow-up prompt asking ChatGPT to spell CurieTech correctly and make the tone funnier and more playful.](../../assets/blog/how-i-use-chatgpt-to-create-articles-4.png) ![Playful, emoji-heavy rewrite titled "CurieTech's Glow-Up: Chat First, Create Later" with a benefits list.](../../assets/blog/how-i-use-chatgpt-to-create-articles-5.png) ![Continuation of the playful rewrite with emoji-laden "Example" and "My Take" sections.](../../assets/blog/how-i-use-chatgpt-to-create-articles-6.png) ![Prompt telling ChatGPT to make the tone "a bit less gay", and its reply offering a more balanced version.](../../assets/blog/how-i-use-chatgpt-to-create-articles-7.png) ![Toned-down rewrite of "CurieTech's Glow-Up: Chat First, Create Later" with a "Why this matters" list.](../../assets/blog/how-i-use-chatgpt-to-create-articles-8.png) ![Continuation of the balanced rewrite with "Example" and "My Take" sections.](../../assets/blog/how-i-use-chatgpt-to-create-articles-9.png) ![Prompt clarifying that Curie is an AI dev tool for MuleSoft projects usable inside Anypoint Studio.](../../assets/blog/how-i-use-chatgpt-to-create-articles-10.png) ![Rewrite grounded in MuleSoft context: "CurieTech's New AI Experience: Chat Before You Build".](../../assets/blog/how-i-use-chatgpt-to-create-articles-11.png) ![Continuation with "Why This Is a Big Deal" and an "Example in Action" using a DataWeave-tests scenario.](../../assets/blog/how-i-use-chatgpt-to-create-articles-12.png) ![The "My Take" section calling Curie a collaborative development partner, plus a follow-up caption offer.](../../assets/blog/how-i-use-chatgpt-to-create-articles-13.png) ![Prompt to keep the "Chat First, Create Later" title, and the polished article reopening with it.](../../assets/blog/how-i-use-chatgpt-to-create-articles-14.png) ![Polished article's "How It Works" and "Why This Matters" sections referencing API type and DataWeave tests.](../../assets/blog/how-i-use-chatgpt-to-create-articles-15.png) ![Polished article's "Example in Action" and "My Take" sections framing Curie as a development partner.](../../assets/blog/how-i-use-chatgpt-to-create-articles-16.png) ![Prompt simply saying "linkedin post", with ChatGPT offering a short scroll-stopping promo post.](../../assets/blog/how-i-use-chatgpt-to-create-articles-17.png) ![Short LinkedIn promo caption "New in CurieTech: Chat First, Create Later" with a newsletter-link placeholder.](../../assets/blog/how-i-use-chatgpt-to-create-articles-18.png) At the end of the day, AI is just a tool—it won’t replace your voice or your perspective. But if you know how to guide it, you can save time, avoid burnout, and still produce content that feels authentic and valuable. The trick isn’t to let AI do the work for you, but to let it work with you. --- ## Implement MySQL with Docker in Anypoint Code Builder (ACB) Source: https://prostdev.com/post/connect-mysql-docker-to-mulesoft-acb-vs-code | Published: Jul 28, 2025 | Category: Tutorials In this tutorial, you’ll learn how to connect a **MySQL database running in Docker** to a MuleSoft project in **Anypoint Code Builder (ACB)** — all inside **VS Code**, without ever leaving your IDE. We’ll use three powerful VS Code extensions: - [Container Tools](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-containers) → run MySQL with a single click - [Database Client](https://marketplace.visualstudio.com/items?itemName=cweijan.vscode-database-client2) → create tables and test queries - [Postcode](https://marketplace.visualstudio.com/items?itemName=rohinivsenthil.postcode) → test your API endpoints By the end, you’ll have a Mule flow that returns live MySQL data in JSON. **Follow up with the GitHub repo**: [https://github.com/alexandramartinez/todo-api-impl](https://github.com/alexandramartinez/todo-api-impl) ## Prerequisites Make sure you already have: - An **Anypoint Code Builder project scaffolded** - **VS Code** with the following extensions installed: - Anypoint Extension Pack - Container Tools - Database Client - Postcode - **Docker Desktop** running ## Step 1: Launch MySQL with Container Tools - In your project folder, create a file called `docker-compose.yml` and paste this in: - Click on the **Run All Services** button inside this file (you have to have the Containers extension for VS Code) ```yaml version: '3' services: db: image: mysql:latest environment: MYSQL_ROOT_PASSWORD: admin MYSQL_DATABASE: mysqldb MYSQL_USER: user MYSQL_PASSWORD: password ports: - 3306:3306 volumes: - db_data:/var/lib/mysql volumes: db_data: ``` ✅ MySQL is now running inside a Docker container. ## Step 2: Connect Using the Database Client Extension - Open the **Database** panel in VS Code. - Click **+ New Connection** and select **MySQL**. - Enter: - **Host**: 127.0.0.1 - **Port**: 3306 - **User**: user - **Password**: password - **Database**: mysqldb - Save and test the connection — you should see it succeed. - Run the following SQL to create your table: ```sql CREATE TABLE tasks( id int NOT NULL PRIMARY KEY AUTO_INCREMENT, title VARCHAR(255) NOT NULL, description VARCHAR(255), dueDate VARCHAR(255), completed BOOLEAN NOT NULL ); ``` ✅ The table is ready — no data inserted yet. ## Step 3: Implement the get-all-tasks Sub-Flow in ACB Now that the database is running and the table is created, let’s implement the flow that will fetch tasks from MySQL. In your scaffolded Mule project, add the following sub-flow to your implementation XML file (for example: `get-all-tasks.xml`): ```xml ``` ### How This Sub-Flow Works - **Start Logger -** Marks the beginning of the flow. This is helpful for debugging and tracking execution in your logs. - **Build the SQL Query -** Uses a DataWeave script (`dw/get-all-tasks-request.dwl`) to generate the SQL query dynamically. This keeps SQL logic separate from the flow. - **Database Select -** Executes the SQL query against MySQL using the connection defined in Mysql_Database_Config. - #[payload] tells Mule to use the SQL string produced by the previous step. - **Transform DB Results to JSON -** Applies a second DataWeave script (`dw/get-all-tasks-response.dwl`) to shape the raw DB results into a clean JSON response for the API. - **End Logger -** Marks the successful completion of the sub-flow. Here are the two DataWeave script files: ```dataweave %dw 2.0 output application/java var completed = attributes.queryParams.completed var dueDate = attributes.queryParams.dueDate var whereClause = if (isEmpty(completed) and isEmpty(dueDate)) "" else "WHERE " ++ ( [ ("completed = $completed") if (!isEmpty(completed)), ("dueDate = $dueDate") if (!isEmpty(dueDate)) ] joinBy " AND " ) --- "SELECT * FROM tasks $whereClause" ``` ```dataweave %dw 2.0 output application/json --- payload map { id: $.id as String, title: $.title, description: $.description, dueDate: $.dueDate, completed: $.completed } ``` ## Step 4: Run and Test with Postcode - Start your Mule app from ACB. - Open the Postcode extension in VS Code. - Create a new request: ``` GET http://localhost:8081/api/tasks ``` - Click Send ✅ You’ll see an empty JSON array ([]) until you insert rows later — but your flow is working and ready for data. ## Final Thoughts You’ve successfully implemented a **MySQL-backed flow in Anypoint Code Builder** using Docker and VS Code extensions. - **Container Tools** handles your database with one click - **Database Client** makes schema management simple - **Postcode** keeps testing inside VS Code ⚡ Next step: Add a **POST /tasks** flow to insert new tasks into your database! ## 🔔 Want More Tutorials? Subscribe to [ProstDev on YouTube](https://www.youtube.com/@prostdev?sub_confirmation=1) for weekly videos on: - Anypoint Code Builder (ACB) - API design and implementation - MUnit testing - MuleSoft best practices ## 🤖 FAQ ### 1. Why should I use Docker for my database in MuleSoft projects? Docker makes it easy to spin up a clean, isolated database environment without needing a manual install. It ensures consistency across development machines and avoids “works on my machine” issues. ### 2. Do I need to know Docker commands to follow this approach? Not necessarily. With the right VS Code extensions (like Container Tools), you can manage containers with simple clicks instead of terminal commands. ### 3. What’s the benefit of using a Database Client extension instead of an external tool like MySQL Workbench? Using a Database Client inside VS Code keeps everything in one place. You can manage schemas, run queries, and debug directly alongside your MuleSoft code. ### 4. How do DataWeave scripts improve database flows? DataWeave lets you dynamically generate SQL queries and format responses. This makes your flows more flexible, avoids hardcoding, and ensures cleaner separation between logic and data. ### 5. Is it safe to build SQL queries dynamically with DataWeave? Yes, as long as you validate and sanitize inputs properly. For production, always guard against SQL injection by using parameters or prepared statements when possible. ### 6. What are common use cases for integrating MuleSoft flows with a database? Typical use cases include CRUD operations for APIs, syncing data between systems, generating reports, or validating user information during API requests. ### 7. How do I test MuleSoft endpoints without leaving VS Code? You can use REST client extensions like Postcode. They let you send HTTP requests and view responses without switching to external tools. ### 8. Why should I separate request and response transformations into different DataWeave files? This improves maintainability. If your SQL logic or JSON response format changes, you only need to update the relevant .dwl file without touching the main flow. ### 9. What’s the advantage of using sub-flows in MuleSoft projects? Sub-flows let you break down complex logic into reusable, modular components. This improves readability, makes testing easier, and allows you to reuse common logic across multiple flows without duplicating code. ### 10. Can I reuse this approach with other databases besides MySQL? Absolutely. You can adapt the same setup for PostgreSQL, Oracle, SQL Server, or any database supported by MuleSoft’s Database connector — just update the Docker image and connector configuration. --- ## Scaffold an API Spec in ACB + VS Code Tips for Beginners Source: https://prostdev.com/post/scaffold-api-spec-anypoint-code-builder-vscode-tips | Published: Jul 21, 2025 | Category: Tutorials If you’ve ever used **Anypoint Studio** to implement an API from a spec, you already know how easy it is to generate flows from a RAML or OpenAPI file. But what about **Anypoint Code Builder (ACB)**? In this guide, I’ll walk you through what happens when you scaffold an API spec in ACB — from clicking **“Implement an API”** to running your Mule app locally. Along the way, you’ll learn essential **VS Code tips**, how to handle `.gitignore`, where your **flows** and **configs** live, and how to switch between **UI and XML views**. > [!TIP] > Whether you're a beginner with MuleSoft or just transitioning to ACB, this walkthrough gives you the confidence to explore and edit scaffolded projects like a pro. ## 🛠️ Step 1: Scaffold the API Spec from Exchange Start with an API spec you’ve published to **Anypoint Exchange** (we used a simple To-Do API in OpenAPI 3.0 format). - Open Anypoint Code Builder inside **VS Code** - Click **Implement an API** - Select your API asset from Exchange - Name your project (e.g., todo-api-impl) - Choose a local folder and hit **Create Project** ACB will automatically generate: - Mule flows for each endpoint - HTTP Listener configuration - Error handling logic (400, 404, 405, 406, 415, etc.) - `mule-artifact.json`, `pom.xml`, .gitignore, and other setup files ## 🧭 Step 2: Explore the Scaffolded Project Here’s what you get out of the box: ### ✅ Auto-generated Flows Each method in your API (GET, POST, etc.) becomes its own flow, wired to the right path. ### 🔀 Error Handling ACB scaffolds global error flows like: - 400 Bad Request - 404 Resource Not Found - 405 Method Not Allowed - and more… You don’t need to define these in the API spec — ACB just adds them. ### 🧩 Configuration Files - `mule-artifact.json`: runtime + project metadata - `pom.xml`: dependencies (e.g., APIkit, HTTP connector) - `log4j2.xml`, embedded sample folders, etc. ## 🔧 Step 3: Use Mule Project Properties UI Right-click your project > **Open Mule Project Properties**. This UI helps you: - View/set Mule runtime and Java version - See which **connectors** were added - Check compatibility for versions (runtime vs connector vs Java) - Apply missing runtimes automatically ## 🗂️ Step 4: Clean Up with .gitignore To keep your repo clean: - Add `src/test/resources/embedded*/` to .gitignore - Ignore .vscode or other machine-specific files - VS Code makes this easy from the Source Control tab — right-click any file > Add to .gitignore ## 🌀 Step 5: Initialize Git & Use Source Control - Open the **Source Control** tab in VS Code - Click “Initialize Repository” - You’ll see all tracked changes - You can commit everything after staging, or one file at a time ## 🧑‍💻 Step 6: Canvas vs XML — Switch Views Easily Want XML to open by default? - Open a .xml file (e.g., a flow) - Click the three-dot icon at the top right > **Configure Editors** - Edit the file association so XML is default instead of Canvas - Switch it back anytime to mulesoft.projectFile.canvas if you prefer UI Bonus: You can even open **both views at once** with "Open in Canvas and Code Editor." ## 🚀 Step 7: Run & Test Your App in VS Code Click the **Run** button or use the command palette to launch the Mule app locally. Test your endpoints with: - Postman - Postcode (VS Code extension for REST requests) - cURL or any HTTP client You should see 200 OK responses — even though the implementation is empty. That’s scaffolding done right. ## 🎯 Recap: What You Learned ✅ How to scaffold an API spec into a running Mule project ✅ What files and flows ACB generates for you ✅ How to explore and customize your project like a pro ✅ Useful VS Code tips for Git, views, testing, and organization ## 🔔 Want More Tutorials? Subscribe to [ProstDev on YouTube](https://www.youtube.com/@prostdev?sub_confirmation=1) for weekly videos on: - Anypoint Code Builder (ACB) - API design and implementation - MUnit testing - MuleSoft best practices ## 🤖 FAQ – Anypoint Code Builder (ACB) API Scaffolding & VS Code ### ❓ What does "scaffold an API" mean in Anypoint Code Builder? Scaffolding an API in ACB means automatically generating Mule flows, error handlers, and configuration files (like `mule-artifact.json` and `pom.xml`) based on an API spec, such as OpenAPI or RAML. This allows developers to instantly start implementing logic without manually creating flows from scratch. ### ❓ How do I implement an API spec from Anypoint Exchange in ACB? You can implement an API by: Opening Anypoint Code Builder in VS Code Clicking **“Implement an API”** Selecting your API spec from Exchange Letting ACB generate flows and configs for you ### ❓ What files are auto-generated when scaffolding an API in ACB? ACB generates: Mule flows for each method Error handlers `mule-artifact.json` `pom.xml` Logging configs (`log4j2.xml`) HTTP listener configurations Sample test files (under `src/main/test/embedded*`) ### ❓ Can I edit the Mule flows in both XML and visual UI (Canvas) in ACB? Yes. ACB supports both views: **Canvas UI** for visual editing **XML editor** for precise config control You can open both views side-by-side or configure VS Code to always default to one view using the **Configure Editor** setting. ### ❓ How do I configure .gitignore in a new ACB project? ACB automatically creates a .gitignore file for you when creating a new project, but you still need to add some files or folders like `src/test/resources/embedded*` or .vscode/ ### ❓ What is mule-artifact.json used for? The `mule-artifact.json` file defines project-level metadata like: The required Mule runtime version Minimum Mule version ACB generates this automatically during scaffolding. ### ❓ How do I run my Mule app locally in ACB? Use the Run/Debug tab in VS Code or use the command palette: Select **Run/Debug Mule Application Locally** Check the terminal for server logs and endpoint availability Use Postman or Postcode (VS Code REST client) to test your API --- ## Design a REST API in Anypoint Code Builder (OpenAPI + VS Code) Source: https://prostdev.com/post/how-to-design-a-rest-api-in-anypoint-code-builder-openapi-vs-code-exchange | Published: Jul 14, 2025 | Category: Tutorials Whether you’re new to MuleSoft or finally starting to explore Anypoint Code Builder (ACB), this post will walk you through designing a full REST API specification using OpenAPI 3.0 — directly inside VS Code, with no XML or flows yet. By the end, you’ll be able to: - Define a reusable OpenAPI spec for a To-Do API - Mock and test the endpoints inside ACB - Publish the spec to Anypoint Exchange so your team can reuse or scaffold flows from it later Let’s build the foundation for your next MuleSoft project — the right way. ## 🛠️ What You’ll Need - New to ACB? Read our [Getting Started guide](https://www.prostdev.com/post/getting-started-with-anypoint-code-builder-in-vs-code-beginner-guide) first. - [Anypoint Platform](https://anypoint.mulesoft.com/) account to publish to Exchange. ## 🧪 What We’re Building We’re designing a To-Do Task Management API that supports full CRUD operations for /tasks: - GET /tasks to list tasks with optional filters - POST /tasks to create a task - GET /tasks/{taskId}, PUT, and DELETE for individual task management - A schema for Task and a separate input model TaskInput > If you want to follow with the YAML file, you can check it out [here](https://gist.github.com/alexandramartinez/c7680921369627323b763905b9b28490). ## 🧑‍💻 Step-by-Step API Design in ACB ### 1. Create a New OpenAPI Spec From the ACB Welcome screen, choose "Design an API", then select: - Type: API Specification - Language: OpenAPI 3.0 (YAML) - Project name: todo-task-api You don’t need to log into Anypoint Platform just yet. ### 2. Fill Out the API Info Block Update the metadata like this: ```yaml info: version: 1.0.0 title: To-Do Task Management API description: A simple API for managing to-do tasks. Supports full CRUD operations for tasks in a single-user context. ``` This info will be displayed in Exchange later — so make it clear and helpful. ### 3. Define Your Endpoints Start with the /tasks resource and support: - GET with optional query parameters (completed, dueDate) - POST with a JSON request body ```yaml paths: /tasks: get: summary: List all tasks description: Retrieve a list of all tasks, with optional filtering by completion status and due date. parameters: - in: query name: completed schema: type: boolean description: Filter tasks by completion status (true or false) - in: query name: dueDate schema: type: string format: date description: Filter tasks by due date (YYYY-MM-DD) responses: "200": description: A list of tasks content: application/json: schema: type: array items: $ref: "#/components/schemas/Task" post: summary: Create a new task description: Add a new task to the to-do list. requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/TaskInput" responses: "201": description: Task created successfully content: application/json: schema: $ref: "#/components/schemas/Task" ``` Then add the /tasks/{taskId} resource with GET, PUT, and DELETE methods. Make sure to handle the URI parameter. ```yaml /tasks/{taskId}: parameters: - in: path name: taskId required: true schema: type: string get: summary: Get a task by ID description: Retrieve a specific task by its unique ID. responses: "200": description: Task details content: application/json: schema: $ref: "#/components/schemas/Task" "404": description: Task not found put: summary: Update a task description: Update an existing task by its ID. requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/TaskInput" responses: "200": description: Task updated successfully content: application/json: schema: $ref: "#/components/schemas/Task" "404": description: Task not found delete: summary: Delete a task description: Remove a task from the list by its ID. responses: "204": description: Task deleted successfully "404": description: Task not found ``` ### 4. Add Reusable Schemas In the components section, define two schemas: #### Task ```yaml components: schemas: Task: type: object properties: id: type: string example: "1" title: type: string example: Buy groceries description: type: string example: Milk, bread, eggs dueDate: type: string format: date example: 2026-01-01 completed: type: boolean default: false example: false required: - id - title - completed ``` #### TaskInput ```yaml TaskInput: type: object properties: title: type: string example: Buy groceries description: type: string example: Milk, bread, eggs dueDate: type: string format: date example: 2026-01-01 completed: type: boolean default: false example: false required: - title - completed ``` ### 5. Test Using the API Console > Your API Specification should look like [this](https://gist.github.com/alexandramartinez/c7680921369627323b763905b9b28490). Click API Console to try out each endpoint — no flows required. [You](http://required.You)’ll see: - Query parameter validation - 404 handling - Schema errors if you send the wrong type - Simulated responses using your example data This is perfect for quick iteration or mocking out APIs. ### 6. Publish to Anypoint Exchange When you're ready to share: - Log into Anypoint Platform via ACB - Press Cmd/Ctrl + Shift + P → "Publish API Project to Exchange" - Choose a business group and version, then publish! You can now discover and reuse this API inside your org — or scaffold flows from it in the next step. ## 📦 What’s Next? In the next tutorial, we’ll take this exact API spec and use it to auto-generate the MuleSoft flows. You’ll implement the endpoints inside VS Code without writing a line of XML manually. 👉 Subscribe to the [YouTube playlist](https://www.youtube.com/playlist?list=PLb61lESgk6hh9SszYvsAnLYiV9-WCpqPP) for new ACB videos every week. ## **💡 Pro Tip** **If you ever get stuck, I offer** **Anypoint Code Builder Office Hours** **to help you troubleshoot and get unblocked.** **👉 Book a session** [here](https://calendly.com/mule-alexmartinez/acb-troubleshooting)**.** --- ## Getting Started with Anypoint Code Builder in VS Code (Beginner Guide) Source: https://prostdev.com/post/getting-started-with-anypoint-code-builder-in-vs-code-beginner-guide | Published: Jul 7, 2025 | Category: Tutorials If you're new to MuleSoft—or making the switch from Anypoint Studio—you might be wondering: **how do I even begin with Anypoint Code Builder (ACB)?** This post will walk you through the **entire setup process of ACB locally in Visual Studio Code**, from installation to building and testing your very first “Hello World” Mule application. Whether you're a developer curious about the future of MuleSoft or a seasoned Studio user adjusting to ACB, this is your go-to beginner’s guide. ## 🛠️ Step 1: Install the Required Tools Before using ACB, make sure you have the following installed: - **Git**: [Download Git here](https://git-scm.com/downloads) - **Visual Studio Code**: [Download VS Code](https://code.visualstudio.com/) ## 🔌 Step 2: Install the Anypoint Extension Pack ✅ **Important:** Only install the Anypoint Extension Pack *after* Git and VS Code are installed and up to date. If needed, uninstall and reinstall to get the latest version of VS Code. - Open VS Code. - Go to the Extensions tab. - Search for Anypoint Extension Pack. - **Install the full pack** (not individual extensions). - Restart VS Code completely after installation. 🔁 If you've already installed extensions one by one, uninstall them, restart VS Code, and then install the full pack. ## 🧠 Step 3: Get Comfortable with VS Code Open the **Command Palette**: - Mac: Shift + Command + P - Windows: Shift + Ctrl + P You’ll use this for everything—from changing themes to running Mule apps. Try searching for: - Preferences: Color Theme Customize your theme or install fun ones like *Halloween* or *MuleSoft Community Theme*. ## 🐴 Step 4: Understand the ACB Layout Unlike Anypoint Studio, ACB runs inside VS Code. Here's how it differs: - **Start a Project** from the MuleSoft tab (not from the “File > New Project” menu). - You can’t import JAR files or projects directly. - Always **open the root folder** (with your src, `pom.xml`, etc.) to work on projects. ### Quick Actions in ACB: - Design an API - Implement an API - Develop an Integration (what we’ll use) ## 🧩 Step 5: Install Additional VS Code Extensions (Optional) VS Code has a powerful extension ecosystem. For example: - Docker - Kubernetes - Postman - MySQL Viewer - GitHub Actions These aren’t required for ACB, but they supercharge your workflow. ## ⚙️ Step 6: Configure ACB Settings In the MuleSoft tab: - Click the gear ⚙️ icon to adjust **autocompletion**, **default runtimes**, and **VM arguments**. - You *don’t need to install Java manually*—ACB does it for you. - In future versions, ACB may also install Git automatically. ## 🔐 Step 7: Log In to Anypoint Platform Click your profile icon or use the “Sign in with Anypoint Platform” option in the ACB panel. (You can skip this for now if you’re just testing locally.) ## 👋 Step 8: Build a "Hello World" Mule App Let’s create our first integration: - Go to the **MuleSoft tab** → “Develop an Integration.” - Name it: helloworld - Choose a folder and select the latest runtime and Java version. - Once the project is created: - Use the UI builder to add an **HTTP Listener**. - Set the path to /hello. - Add a **Transform Message** component and output "Hello World". - Add a **Logger** to print the payload. You now have a fully working Mule app that returns “Hello World” at [http://localhost:8081/hello](http://localhost:8081/hello). ## 🐞 Step 9: Run and Debug Your App - Use the **Run and Debug** panel in VS Code. - Click “Start Debugging.” - Test with a tool like Postcode (or Postman). - Hit [localhost:8081/hello](http://localhost:8081/hello) to get your response. ### Debugging Tips: - Add breakpoints directly in the XML or use the UI. - Watch variables like payload or check flow attributes. - Use the Command Palette to format XML files with Format Document. ## 🧪 Final Thoughts You’ve now: ✅ Installed ACB ✅ Created a Hello World Mule App ✅ Tested and Debugged it—all inside VS Code! This is just the beginning. In the next tutorial, we’ll build a real REST API using API specifications and flow logic. ## 💡 Pro Tip If you ever get stuck, I offer **Anypoint Code Builder Office Hours** to help you troubleshoot and get unblocked. 👉 Book a session [here](https://calendly.com/mule-alexmartinez/acb-troubleshooting). --- ## Building a YouTube MCP Server with MuleSoft and CurieTech AI in 10 Minutes Source: https://prostdev.com/post/building-a-youtube-mcp-server-with-mulesoft-and-curietech-ai-in-10-minutes | Published: Jun 19, 2025 | Category: Tutorials Curious how to integrate the YouTube Data API v3 into a fully functioning MCP (Model Context Protocol) server in record time? With the help of CurieTech AI and MuleSoft, I did it in just 10 minutes, without ever having created an MCP server before. In this article, I’ll walk you through what I did step-by-step — including how I used CurieTech to generate the entire integration, converted it into an MCP server, and ran it using SuperGateway and the MCP Inspector. I also show how you can connect this to Claude or other apps and use natural language to run YouTube searches via the MCP server. 🔗 **GitHub Repo:** [https://github.com/alexandramartinez/youtube-mcp-server-mule](https://github.com/alexandramartinez/youtube-mcp-server-mule) ## 🧠 What Is MCP? MCP stands for **Model Context Protocol**, a new protocol designed for agents, AI tools, and other automated systems to interact with structured tools in a flexible and self-describing way. MCP tools define schemas, input parameters, and responses — and are ideal for agent-based use cases. > In other words... It's an API for your agents. ## 🛠️ What We’re Building A MuleSoft app that: - Connects to the **YouTube Data API v3** - Accepts search queries and API keys as parameters - Returns YouTube search results - Runs as an **MCP server** with full schema and parameter validation ## 🔑 Step 1: Get Your YouTube API Key To use the YouTube API, you'll need an API key from Google Cloud: - Go to [Google Cloud Console](https://console.cloud.google.com/). - Create a project. - Enable the **YouTube Data API v3**. - Under **APIs & Services > Credentials**, create a new **API Key**. - Under **API Restrictions**, select **YouTube Data API v3**. - Save your changes and copy the key — you'll use it in your requests. ## 🧪 Step 2: Generate the API with CurieTech AI Inside [CurieTech AI](https://curietech.ai): - Go to the **Integration Generator**. - Set the language to **Java 17**, build tool to **Maven 3.9**, and **Mule 4.9**. - Use a prompt like: "Make an API that uses YouTube Data API v3. Include all query parameters, especially key, and make it as simple as possible." - Paste the list of parameters from the [official docs](https://developers.google.com/youtube/v3/docs/search/list) or paste the URL. - CurieTech generates the full integration — copy the code. ## 💻 Step 3: Run the App in MuleSoft - Create a new Mule project in **Anypoint Code Builder** or **Studio**. - Paste the code from CurieTech. - Run the project (default port is 8081). - Use Postman to test: ``` GET http://localhost:8081/search?key=YOUR_API_KEY&q=mulesoft ``` ## 🤖 Step 4: Convert to an MCP Server - Push your Mule project to GitHub (or use VS Code's repo initializer). - In CurieTech, open the **Coding Agent** and select your repo. - Prompt: "Turn this into an MCP server using the beta MCP connector." - CurieTech updates the code, modifies the `pom.xml`, adds a global MCP config, and replaces the HTTP listener with an MCP listener. - Approve the PR in GitHub and merge. ## 🚦 Step 5: Run SuperGateway - Open a terminal window. - Run the following command (ensure the port matches your Mule app's HTTP Listener): ```bash npx -y supergateway --sse http://localhost:8081/sse --ssePath /sse --messagePath /message ``` ## 🔍 Step 6: Open MCP Inspector - Open a new terminal window and run: ```bash npx @modelcontextprotocol/inspector ``` - Copy the tokenized URL (e.g., http://localhost:6274/?MCP_PROXY_AUTH_TOKEN=...) into your browser. - Select SSE as the transport type - Paste the following URL: ``` http://0.0.0.0:8081/sse ``` - Click Connect - Select the youtube-search tool - Enter your API Key and query - Click on Run Tool ## 🤝 Step 7: Add the Server to Your App (e.g., Claude) Add the following to your Claude or other AI agent config: ```json { "mcpServers": { "youtube": { "command": "npx", "args": [ "-y", "supergateway", "--sse", "http://0.0.0.0:8081/sse", "--ssePath", "/sse", "--messagePath", "/message" ] } } } ``` Restart your app, and you're ready to use it in natural language prompts! ## 🧠 Example Prompts You Can Use Inside Claude (or any compatible tool): - “Search for the ProstDev YouTube channel and return the last 4 videos.” - “Generate a LinkedIn post that promotes the first video. Include the title, description, and video link.” - “Find trending content about ‘API design’ using YouTube search.” ## 📦 Repo 🔗 **GitHub Repo:** [https://github.com/alexandramartinez/youtube-mcp-server-mule](https://github.com/alexandramartinez/youtube-mcp-server-mule) ## 🧭 Final Thoughts I had never worked with MCP before — and I built this whole server in 10 minutes thanks to **CurieTech AI**. The entire flow, from integration to deployment and testing, was smoother than I imagined. If you learn by reading code like I do, this is one of the fastest ways to master new patterns like MCP. Whether you're building for fun, research, or AI-driven platforms, give this a try and let me know what you build next! --- ## Why is my Array not filtering correctly? It may be an Array instead Source: https://prostdev.com/post/array-string-key-dataweave | Published: Feb 5, 2025 | Category: Tutorials ## The Problem Ok so here's what was happening to me and I didn't know why. I was doing a bunch of transformations and at one point I'd add some keys and/or values into an array. So, some of the arrays were of type `Array` (because no key was added here) while some of them where `Array` I didn't think much about it at first, since I would -incorrectly- see all the arrays in the JSON output were of type `Array`. Like this: > [!NOTE] > The item in line 4 was coerced into a Key for demonstration purposes. ```dataweave title="transform.dwl" %dw 2.0 output application/json var arr1 = [ "key" as Key, "string" ] var arr2 = [ "key", "string" ] --- [arr1, arr2] ``` ```json title="output.json" [ [ "key", "string" ], [ "key", "string" ] ] ``` > [!PLAYGROUND] > [Open in Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=ProstDev%2Fdataweave-playground-previews&path=scripts%2Farray-string-key%2Fscript1) So imagine my surprise when I tried to use a [distinctBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-distinctby) function with the above arrays and I'd still get the same two arrays in the output. ```dataweave title="transform.dwl" %dw 2.0 output application/json var arr1 = [ "key" as Key, "string" ] var arr2 = [ "key", "string" ] --- [arr1, arr2] distinctBy $ ``` ```json title="output.json" [ [ "key", "string" ], [ "key", "string" ] ] ``` > [!PLAYGROUND] > [Open in Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=ProstDev%2Fdataweave-playground-previews&path=scripts%2Farray-string-key%2Fscript2) I started losing my mind trying to troubleshoot the problem a million different ways because I didn't notice the items in the arrays were of different types! It was until one of those million times finally led me to see the array types were actually different 🤦 ## Possible Solutions ### Solution 1 There are different solutions for this problem. One would be to make sure any new items are of type String and not Key. This completely removes the issue since now you'd be comparing two arrays of type `Array`, which correctly returns only one array in the output. ```dataweave title="transform.dwl" %dw 2.0 output application/json var arr1 = [ "key" as Key, "string" ] var arr2 = [ "key", "string" ] --- [arr1, arr2] map ((item) -> item map ($ as String) ) distinctBy $ ``` ```json title="output.json" [ [ "key", "string" ] ] ``` > [!PLAYGROUND] > [Open in Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=ProstDev%2Fdataweave-playground-previews&path=scripts%2Farray-string-key%2Fscript3) ### Solution 2 If you can't change the type of the items or it would take too many iterations to do, another thing you can do is to transform the items inside the [distinctBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-distinctby) function itself. Like this: ```dataweave title="transform.dwl" %dw 2.0 output application/json var arr1 = [ "key" as Key, "string" ] var arr2 = [ "key", "string" ] --- [arr1, arr2] distinctBy ($ joinBy ",") ``` ```json title="output.json" [ [ "key", "string" ] ] ``` > [!PLAYGROUND] > [Open in Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=ProstDev%2Fdataweave-playground-previews&path=scripts%2Farray-string-key%2Fscript4) Since we're using the [joinBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-joinby) function, we're transforming the values of arr1 and arr2 into concatenated strings and THEN comparing them. Both arr1 and arr2 would be "key,string", which makes these values the same. I hope this post brings you more DataWeave knowledge or helps you fix some code :D Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## How to connect your local Ollama AI using the MuleSoft AI Chain (MAC) project and Anypoint Code Builder (ACB) Source: https://prostdev.com/post/mac-project-ollama-ai | Published: Nov 20, 2024 | Category: Tutorials This post guides you through building an AI-powered API using Ollama and MuleSoft. You'll start by installing Ollama locally, designing and implementing the API, and testing it in your development environment. With ngrok, you'll expose the API publicly, and finally, deploy it to CloudHub for production use. For guidance, you can follow along using [this GitHub repository](https://github.com/ProstDev/mac-ollama-proj). ## 1 - Install Ollama locally - Go to [ollama.com](http://ollama.com) and follow the prompts to install in your local computer - Make a note of which version you started running (like llama3 or llama3.2) - To verify ollama is running, you should either be able to interact with it in the terminal or you should be able to run "ollama list" or "ollama ps" - We will verify the local installation in MuleSoft before deploying to CloudHub ![Terminal running ollama run llama3, pulling the model and answering a sample prompt](../../assets/blog/mac-project-ollama-ai-1.png) ## 2 - Create the API specification - Open Anypoint Code Builder > Design an API - Name it MAC-Ollama-API - Select REST API > RAML 1.0 - **Create Project** - Paste the following into this RAML file ```yaml #%RAML 1.0 title: MAC-Ollama-API version: 1.0.0 description: Simple API to connect a Mule application and Ollama locally using the MAC project. mediaType: application/json /chat: post: body: description: The question to ask Ollama type: string example: "Hello!" responses: 200: description: The response from Ollama body: type: string example: Hey! How's it going? ``` - Publish to Exchange using the Command Palette in VS Code (cmd+shift+P or ctrl+shift+P) ![VS Code Command Palette showing the "Publish API Specification to Exchange" command](../../assets/blog/mac-project-ollama-ai-2.png) - You can leave all the defaults (organization, project name, version, etc.) and wait for it to be published ## 3 - Implement the API - Select **Yes** to implement this API in ACB - Name it mac-ollama-proj, select the folder where you want to keep this, select Mule Runtime 4.8 and Java 17 - Once the project finishes loading, open the `mac-ollama-proj.xml` file under `src/main/mule` - Click on the **Flow List** button and switch to the one that starts with "post" ![Anypoint Code Builder Flow List dropdown listing the project's flows, including the post flow](../../assets/blog/mac-project-ollama-ai-3.png) - Remove the logger from the flow - Click on the plus button to add a new connector - Click on the Exchange button to search in Exchange ![Add Component panel with the Search in Exchange button highlighted, listing connectors](../../assets/blog/mac-project-ollama-ai-4.png) - Search for the MuleSoft AI Chain module and click on it - Select the **Chat answer prompt** connector ![MuleSoft AI Chain Connector operations list with Chat answer prompt selected](../../assets/blog/mac-project-ollama-ai-5.png) - Click on the new connector from the canvas - Click on the plus button next to the Connection Config ![Chat answer prompt connector config panel with the Add Connector Config plus button](../../assets/blog/mac-project-ollama-ai-6.png) - Add the following details: | Setting | Value | Notes | |---|---|---| | Name | `MAC_Config` | | | LLM type | OLLAMA | | | Config type | Configuration Json | | | File path | `mule.home ++ "/apps/" ++ app.name ++ "/llm-config.json"` | *Make sure to click on the fx button to make this a formula instead of a hardcoded value | | Model name | llama3 | *This will depend on the model you installed locally | | Temperature | 0.7 | | | LLM timeout | 60 | | | LLM timeout unit | SECONDS (Default) | | | Max tokens | 500 | | > [!NOTE] > This post was created when the MAC project was on version 1.0.0. This functionality may change in future versions. - Click Add - Back in the canvas, click on the plus button at the end to add a new connector - Add a Transform - Set it up like the following: ```dataweave output application/json --- payload ``` ![Transform component after Chat answer prompt, set to output application/json payload](../../assets/blog/mac-project-ollama-ai-7.png) > [!NOTE] > If you see a red line/dot at the beginning of the script like in the previous screenshot, head to the XML view and remove the surrounding #[ ] - You can add some loggers at the beginning and/or at the end of the flow to show the input/output payloads from the console - The Global Configuration / MAC Connection Config should look like the following (located at the beginning of the XML file): ```xml ``` - The flow should look like the following (located at the end of the XML file): ```xml ``` - Create a new file called `llm-config.json` under `src/main/resources` and paste the following: ```json { "OLLAMA": { "OLLAMA_BASE_URL": "http://localhost:11434" } } ``` ## 4 - Test the app locally - Run the application locally - Once it has started, send a POST request to [localhost:8081/api/chat](http://localhost:8081/api/chat) with a JSON body including your question ![API client POST to localhost:8081/api/chat returning a 200 OK chat response from Ollama](../../assets/blog/mac-project-ollama-ai-8.png) - If everything was successful, continue to the next step. Otherwise, please troubleshoot before continuing - Stop the app ## 5 - Use ngrok for the public endpoint - Download and install [ngrok](https://download.ngrok.com/mac-os) to make your Ollama endpoint publicly available from the internet. This way, CloudHub will be able to access the URL since it's not only in your local (localhost:11434) - Run the following from your Terminal: ```bash ngrok http 11434 --host-header="localhost:11434" ``` - Copy the address from the Forwarding field ![ngrok session screen with the public Forwarding URL pointing to localhost:11434 highlighted](../../assets/blog/mac-project-ollama-ai-9.png) - Paste it in your `llm-config.json` file under `src/main/resources` ![llm-config.json with OLLAMA_BASE_URL set to the ngrok forwarding URL](../../assets/blog/mac-project-ollama-ai-10.png) - Save the file and run the app again to verify everything still works correctly - Stop the app once you verify it works ## 6 - Deploy to CloudHub - If everything still works, you are ready to deploy to CloudHub - Go to your `pom.xml` file and change the version to 1.0.0 (remove the SNAPSHOT) ![pom.xml version changed to 1.0.0 with the SNAPSHOT suffix removed](../../assets/blog/mac-project-ollama-ai-11.png) - Save the file - Head to your `mac-ollama-proj.xml` file and click on the Deploy to CloudHub button ![mac-ollama-proj.xml editor toolbar with the Deploy to CloudHub button highlighted](../../assets/blog/mac-project-ollama-ai-12.png) - Select CloudHub 2.0 - Select the `US-EAST-2` (or whichever region is available on your account) - Select the Sandbox environment - Make sure everything looks good in the newly created `deploy_ch2.json` file and click on Deploy - Select the Runtime version (including the patch) - It will first publish the project to Exchange (as an application) - After that, it will be deployed to CloudHub - If you're experiencing a lot of issues deploying from ACB, you can also take the generated JAR from the target/ folder and deploy it manually to Runtime Manager - Once the deployment is done, get the Public Endpoint from your application and call it to verify the app works ![API client POST to the deployed CloudHub endpoint returning a 200 OK chat response](../../assets/blog/mac-project-ollama-ai-13.png) Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## How to develop AsyncAPIs on Anypoint Code Builder with Solace PubSub+ locally (Docker) Source: https://prostdev.com/post/how-to-develop-asyncapis-on-anypoint-code-builder-with-solace-pubsub-locally-docker | Published: Nov 7, 2024 | Category: Tutorials First of all, I am brand-new to any Solace stuff. So there I was, working on a demo to connect Mule with [Solace](https://solace.com/) PubSub+ (Message Broker) but I didn't want to create a 30 or 60-day cloud trial account with them because I wanted to be able to re-run this demo as many times as needed. So, I saw you could run the broker using Docker. The problem is that all the documentation from the MuleSoft side were for Solace in the cloud and not local. So, here I am creating this post for you! If you want to run Solace locally, using their Docker image/container, and then connect that to a MuleSoft AsyncAPI specification/application -- then, keep reading. > [!NOTE] > Take a look at the following GitHub repo to follow along with the code: > [asyncapis-accounts-email](https://github.com/alexandramartinez/asyncapis-accounts-email) ## 1 - Start the Solace Docker Container I started by following the [Getting Started with PubSub+ Standard Edition](https://solace.com/products/event-broker/software/getting-started/) page, which pretty much gives you the command to run to start the new Docker container: ```bash docker run -d -p 8080:8080 -p 55554:55555 -p 8008:8008 -p 1883:1883 -p 8000:8000 -p 5672:5672 -p 9000:9000 -p 2222:2222 --shm-size=2g --env username_admin_globalaccesslevel=admin --env username_admin_password=admin --name=solace solace/solace-pubsub-standard ``` > [!NOTE] > The command above is used for Mac only. Please refer to the website for the Linux/Windows command. However, I had one issue with the port 9000, so I had to change that to map to the port 9001 instead. If you have a port issue, you can also change it. Make sure you change the port at the **left** side and not the one at the right side: ```bash : ``` The other thing you'll notice from the website is that the commands are different depending on the operating system you're running. It states: *Your applications can use Solace APIs to connect to the message broker on this port. Note that for Mac users we are mapping port 55554 to the default Solace SMF port of 55555 because Mac OS now reserves port 55555.* Anyway, long story short, if you can run the command from the website without errors, good. If not, modify it to match your available ports. For example, since my port 9000 was busy (and I'm running Mac), this is the command I had to use: ```bash docker run -d -p 8080:8080 -p 55554:55555 -p 8008:8008 -p 1883:1883 -p 8000:8000 -p 5672:5672 -p 9001:9000 -p 2222:2222 --shm-size=2g --env username_admin_globalaccesslevel=admin --env username_admin_password=admin --name=solace solace/solace-pubsub-standard ``` Once your container is running, you should be able to see a green icon next to it from your Docker Desktop application. ![Docker Desktop Containers list with the running solace container and a green status dot](../../assets/blog/how-to-develop-asyncapis-on-anypoint-code-builder-with-solace-pubsub-locally-docker-1.png) ## 2 - Set up the queue Once everything looks good, open your browser and head to [http://localhost:8080/](http://localhost:8080/). This will open a log-in screen to connect to Solace's management tool. If you're not seeing this screen immediately, wait a few minutes. ![Solace PubSub+ Standard login screen with the admin username filled in](../../assets/blog/how-to-develop-asyncapis-on-anypoint-code-builder-with-solace-pubsub-locally-docker-2.png) Write the credentials admin/admin and log in. Once inside, click on the default VPN. ![Solace Message VPNs screen showing the default VPN card with status Up](../../assets/blog/how-to-develop-asyncapis-on-anypoint-code-builder-with-solace-pubsub-locally-docker-3.png) From the left side, click on Queues and add a new Queue by clicking on the green **+Queue** button on the top right. Write the name of your queue and leave all the defaults. For our example, we'll use the name user-signup. ![Solace Queues tab with a new user-signup queue and the +Queue button](../../assets/blog/how-to-develop-asyncapis-on-anypoint-code-builder-with-solace-pubsub-locally-docker-4.png) Click on this new queue to open its properties and head to the **Subscriptions** tab at the top. Add a new subscription/topic with the same name (user-signup). ![user-signup queue Subscriptions tab with a user-signup topic subscription added](../../assets/blog/how-to-develop-asyncapis-on-anypoint-code-builder-with-solace-pubsub-locally-docker-5.png) You will need to do this configuration in order to receive the messages from MuleSoft. ## 3 - Create the AsyncAPI specifications Head to Anypoint Code Builder and create a new AsyncAPI specification called **Accounts Service**. We will be using version 2.6 YAML for this example. Paste the following spec. ```yaml asyncapi: '2.6.0' info: title: Account Service version: '1.0.0' description: Publishes the UserSignedUp event when a new user account is created. contact: name: Alex Martinez email: alexandra.martinez@salesforce.com url: alexmartinez.ca license: name: test url: test defaultContentType: application/json tags: - name: solace description: makes use of Solace PubSub+ servers: Solace: protocol: solace bindings: solace: msgVpn: default url: localhost:55554 channels: user-signup: subscribe: description: Publishes the UserSignedUp event operationId: emitUserSignUpEvent message: $ref: "#/components/messages/UserSignedUp" components: messages: UserSignedUp: payload: type: object properties: firstName: type: string examples: - Alex lastName: type: string examples: - Martinez email: type: string examples: - fake@email.email createdAt: type: string examples: - 2024-01-01T15:00:00Z ``` Note that we are using the **default** VPN and the `localhost:55554` URL. From the first step, the installation, you can see which port you're exposing locally to make use of port 55555 in Docker. In my case, since I'm using a MacOS, my port was changed to be 55554:55555. If you're using Linux/Windows, you probably set it up as 55555:55555. If this is the case, then please change the URL to match your port (the one on the left of the colon - 55555:). Publish this spec to Exchange and create a new one called **Email Service**. Paste the following code. ```yaml asyncapi: '2.6.0' info: title: Email Service version: '1.0.0' description: Subscribed to receive the UserSignedUp event to send the new user a welcome email. contact: name: Alex Martinez email: alexandra.martinez@salesforce.com url: alexmartinez.ca license: name: test url: test defaultContentType: application/json tags: - name: solace description: makes use of Solace PubSub+ servers: Solace: protocol: solace bindings: solace: msgVpn: default url: localhost:55554 channels: user-signup: publish: description: Subscribed to receive the UserSignedUp event operationId: onUserSignUp message: $ref: "#/components/messages/UserSignedUp" components: messages: UserSignedUp: payload: type: object properties: firstName: type: string examples: - Alex lastName: type: string examples: - Martinez email: type: string examples: - fake@email.email createdAt: type: string examples: - 2024-01-01T15:00:00Z ``` Same as before, make sure your VPN and URL match what you set up in Docker. Publish it to Exchange. > [!DOCS] > For more information about why we're using these two specifications, see [Design AsyncAPI Specifications With a Practical Example](https://blogs.mulesoft.com/dev-guides/how-to-design-asyncapi-specifications/). Video version can be found at the end of that article. ## 4 - Develop the Mule apps From Anypoint Code Builder, select the **Implement an API** option. Search for the **Accounts Service** specification you published to Exchange and start implementing it. Take a look at the `global-configs.xml` file. Since we are running Solace locally, there's no need to use the OAuth configuration. We can comment this out. ![global-configs.xml in ACB with the Solace OAuth configuration block commented out](../../assets/blog/how-to-develop-asyncapis-on-anypoint-code-builder-with-solace-pubsub-locally-docker-6.png) Apart from that, let's just add an HTTP Listener connection. ```xml ``` Now head to the `flows.xml` file. Add a flow with an HTTP Listener and a Publish operation (from the APIKit AsyncAPI module). You can copy and paste the following XML code. ```xml ``` Finally, head to the `dev-properties.properties` file. Add the **default** value for the username and password properties. Since we won't be using OAuth, you can comment out the clientId, tokenProviderUrl, and clientSecret properties. ![dev-properties.properties with default username/password and OAuth properties commented out](../../assets/blog/how-to-develop-asyncapis-on-anypoint-code-builder-with-solace-pubsub-locally-docker-7.png) Once you're done with the Accounts Service application, open a new VS Code window and create the Email Service application (based on the Email Service specification we just published to Exchange). Change the properties and remove the OAuth configuration from the `global-configs` file just as we did with the Accounts app. In this case, you don't need to change anything in the `flows` file. ## 5 - Run apps and test locally Run both projects locally. Since ACB doesn't allow this, as workaround, change the Mule runtime version from one of the projects to a different one. For example, accounts with runtime 4.8 and email with runtime 4.7. You can do this by right-clicking any of the files under `src/main/mule` and selecting **Project Properties**. ![ACB file explorer right-click menu with the Project Properties option highlighted](../../assets/blog/how-to-develop-asyncapis-on-anypoint-code-builder-with-solace-pubsub-locally-docker-8.png) ![Project Properties pane setting the Mule Runtime to 4.8.0 and Java to 17](../../assets/blog/how-to-develop-asyncapis-on-anypoint-code-builder-with-solace-pubsub-locally-docker-9.png) Once both projects are running at the same time on your local, open any REST application like Postman and send a request to localhost:8081/publish with the following JSON body: ```json { "firstName": "Alex", "lastName": "Martinez", "email": "fake@email.email", "createdAt": "2024-01-01T15:00:00Z" } ``` You should be able to see this payload from the Email app's console in ACB. ## Troubleshooting I had to do some troubleshooting because it wasn't clear to me which port I should be using to connect to my Solace broker. [This post](https://stackoverflow.com/questions/50641285/solace-connectivity-issue-using-spring-4-x) in StackOverflow was incredibly helpful! I also discovered that MuleSoft is connecting to Solace via [JCSMP](https://docs.solace.com/API/Messaging-APIs/JCSMP-API/jcsmp-api-home.htm) - it makes a LOT of sense since Mule is Java/SpringBoot under the hood, it just wasn't obvious from reading the main docs. Anyway, what I gathered from the StackOverflow post was that, in case your port (55555) was modified for whatever reason, you could double-check it with the following commands: - On a regular terminal in your local (not in Docker), run *docker exec -it solace /usr/sw/loads/currentload/bin/cli -A* - This connects to the Docker container using the Solace CLI. Then, execute *show service* - Press any key to continue You should see an output like the following: ![Solace CLI "show service" output listing the SMF over TCP port as 55555](../../assets/blog/how-to-develop-asyncapis-on-anypoint-code-builder-with-solace-pubsub-locally-docker-10.png) And from the StackOverflow post, "You are looking for the SMF over TCP port. The default port number is 55555." And as you can see from the previous image, it indeed is port 55555. If you have a different port, you will see a different number here. Then, you will just have to check which local port is exposing *that* port from Docker and use that one instead. That's all for this article! I hope this helped guide you to run Solace PubSub+ locally and understand how to run it from MuleSoft! It took me a while to understand it so I really hope if helps someone. Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## How to upsert fields from an object in an array with the update operator in DataWeave Source: https://prostdev.com/post/upsert-object-from-array | Published: Sep 16, 2024 | Category: Tutorials In this post, we'll learn how to use the update operator along with the upsert and conditional options. We'll also learn different ways of handling null values for our fields. > [!TIP] > You can follow along by clicking on the "Open in Playground" buttons to see how the code is being developed and to try it out yourself! ## Use case Let's say you have an array of objects `Array` like so: ```json [ { id: 1, name: "alex1", notes: "NA" }, { id: 2, name: "alex2", notes: "NA" }, { id: 3, name: "alex3", notes: "NA" } ] ``` And you want to update one of those objects depending on the ID. For example, using this variable: ```dataweave var updateUser = { id: 2, notes: "this is my new note", newField: "abc" } ``` The first observations we can make, based on this information, are: - The field `name` is not available from the `updateUser` variable. If we don't want to lose this field, we can't just replace the whole object. We'll need to check each field separately. - The field `newField` is not available from the original user. We will need to insert this field and not just update the current user. A quick solution to this problem is to use the [update operator](https://docs.mulesoft.com/dataweave/latest/dw-operators#update-operator) because it has an upsert option to it. Let's see how to do this. ## Solution First of all, we need to find in the array the user we want to update. To do this, we'll use a [map](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-map) function and a conditional based on the ID field: ```dataweave users map ((user) -> if (user.id ~= updateUser.id) ??? else user ) ``` > [!PLAYGROUND] > [Open in Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=ProstDev%2Fdataweave-playground-previews&path=scripts%2Fupsert-object-from-array%2Fscript1) Once we meet the condition, we can get started with the `update` operator. Since we want to have the possibility of updating all the fields (except the ID), we'll have to create `case`s for all of these. Like so: ```dataweave users map ((user) -> if (user.id ~= updateUser.id) user update { case na at .name -> updateUser.name case no at .notes -> updateUser.notes case nf at .newField -> updateUser.newField } else user ) ``` > [!PLAYGROUND] > [Open in Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=ProstDev%2Fdataweave-playground-previews&path=scripts%2Fupsert-object-from-array%2Fscript2) In summary, the object we want to update has two issues that we need to solve: ```json { "id": 2, "name": null, "notes": "this is my new note" } ``` - The field `name` now has a `null` value - The field `newField` is not being inserted yet Issue 1 is happening because we are updating the field `name` with this: `updateUser.name` - and since this field does not exist, it's returning a `null` value. The solution is quite simple: we need to add a `default` keyword with the original value, like so: ```dataweave case na at .name -> updateUser.name default na ``` Now it will take the value from the original object and use that if the `updateUser` variable does not contain this value. We can add this to the other fields just in case. Issue 2 is where we will be using our `upsert` operator because we do not have this field in our original value, so, we have to tell the operator that we want it to insert it if it doesn't exist. We do that by adding an exclamation point (`!`) right next to the field. Like so: ```dataweave case nf at .newField! -> updateUser.newField ``` > [!PLAYGROUND] > [Open in Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=ProstDev%2Fdataweave-playground-previews&path=scripts%2Fupsert-object-from-array%2Fscript3) Ta-da! 🎉 Now we are correctly inserting this new value and updating the other fields from the original object. ## Handling null values What would happen though, if the `newField` is not available in either of the two objects? Not in the original and not in the `updateUser` variable. In that case, the `update` operator will still add the field because you are using the `upsert` operator. It will just be added with a `null` value. ```dataweave "newField": null ``` Depending on your business requirements, maybe this is the way it should work. But in case it's not, you can fix it two ways: adding a conditional to each field or adding a `skipNullOn` configuration to the output directive. ### With a conditional This is the less fancy approach, but you may need to choose this if you only need to remove specific fields and not ALL the fields with null values. ```dataweave case nf at .newField! if(!isEmpty(updateUser.newField default nf)) -> updateUser.newField default nf ``` > [!PLAYGROUND] > [Open in Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=ProstDev%2Fdataweave-playground-previews&path=scripts%2Fupsert-object-from-array%2Fscript4) > [!TIP] > You can create a local scope with [do](https://docs.mulesoft.com/dataweave/latest/dataweave-flow-control#control_flow_do) if you don't want to re-type `updateUser.newField default nf` twice. ### With skipNullOn This will get rid of all the fields that have a null value. It's super effective if that's what you're looking for, but it may be confusing if later on you're expecting to see actual null values from other fields. To add this, you have to change the output directive on the top of your DataWeave script. Like so: ```dataweave output application/json skipNullOn="everywhere" ``` > [!PLAYGROUND] > [Open in Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=ProstDev%2Fdataweave-playground-previews&path=scripts%2Fupsert-object-from-array%2Fscript5) > [!DOCS] > To learn more about the JSON writer properties, see [JSON Format](https://docs.mulesoft.com/dataweave/latest/dataweave-formats-json#writer_properties). There are a lot more ways to personalize your DataWeave scripts. These are just a few options that you can use to make the best of the operators available! I hope you found this article useful. Add a comment if you found an easier way to do this! ;) Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## Part 5: Data Cloud + MuleSoft integration - Insert data with the BULK operations Source: https://prostdev.com/post/part-5-data-cloud-mulesoft-integration | Published: Jul 30, 2024 | Category: Tutorials Hello, hello! By this point, you should already have a full Mule application running in CloudHub and you have been able to call the Query/Streaming operations. In July 2024, I released a new JAR (version 2.1.0) where I added the Bulk operations. Let's take a look at what other updates have been done to this version! ## Prerequisites - **Previous configurations** - Make sure you followed at least parts 1-3 before reading this article. Part 4 is preferred for security reasons, but not necessary. You will need to do the Data Cloud, MuleSoft, and Postman configuration before doing this post. - **Deployed Mule app -** Make sure you're using the JAR version 2.1.0 or newer. You can find the releases [here](https://github.com/alexandramartinez/datacloud-mulesoft-integration/releases/). - **Postman collection -** If you have already imported the previous Postman collection, make sure you re-import the latest one (released on July 2024) from [this link](https://github.com/alexandramartinez/datacloud-mulesoft-integration/tree/main/rest-clients-requests/postman). This collection includes both the Streaming and the Bulk operations. Below is a screenshot of what the new collection looks like. ![Postman Data Cloud Integration collection listing the streaming and new bulk requests](../../assets/blog/part-5-data-cloud-mulesoft-integration-2.png) ## Understanding the Bulk process Before jumping right away into the new operations, let's take a moment to understand how Data Cloud processes this type of data. Take a look at the following diagram: ![Diagram of the bulk Job lifecycle: create, upload, close, then Data Cloud's InProgress/JobComplete states](../../assets/blog/part-5-data-cloud-mulesoft-integration-3.png) The blue circles are the actual operations you can do with the Data Cloud connector in MuleSoft and the purple squares are what Data Cloud changes on its own. A happy path processing would go like this: - Create the Job - Upload the data to the Job - Close the Job with the *UploadComplete* status - Data Cloud queues this data and changes the Job's state to *InProgress* - Once the data is processed, the Job's status is changed to *JobComplete* The Aborted state would happen if you wanted to stop the processing of the Job after it's already been created. It is irrelevant if you already uploaded the data or not. Once you abort it, the data will be deleted from the Job. The Failed state would happen if there was an error processing the Job in Data Cloud. For example, if the data you sent is corrupted, wrong, or incomplete. You can always delete a Job at any point while the job is not being processed and is already closed. This means after you close it (Aborted/UploadComplete state) but before it is queued (not in the InProgress state), or after it has been processed (Failed/JobComplete). If a specific object already contains an open/in progress Job, you won't be able to create more Jobs for the same object and you will receive a 409-Conflict error when you try to create a new Job. In this case, you will need to wait until the Job is Failed/JobComplete, or you will have to abort the other Open Job first (if it hasn't been closed). A lot of this process is already taken care of by the Mule application in the JAR file. You only need to send your data to the upsert operation and the app will take care of creating, uploading, and closing the Job for you. ## Get All Jobs Once you import the Postman collection, you will see a request called "get all jobs". This is a GET request calling the path `/api/bulk` of our Mule application. You don't need to add Query Params or a body. The credentials you set up in CloudHub are good enough to perform the call. This request will return all the Jobs with their corresponding information, like the ID and its state. *Note that the next screenshot has some blurred information for security reasons. ![Postman "get all jobs" GET returning 200 OK with a JSON array of Jobs and their states](../../assets/blog/part-5-data-cloud-mulesoft-integration-4.png) ## Upsert Once you import the Postman collection, you will see a request called "upsert (CSV)" and another one called "upsert (JSON)". Both are the same request, calling a POST in `/api/bulk/upsert`, but you have the possibility to send different bodies on each. Internally, the Mule application is transforming the input body to a CSV that can be read by Data Cloud. You need to add two Query Parameters, same as you did for the streaming insertion: - `sourceApiName` - `objectName` This request will create a new Job, upload the data, and close it with the *UploadComplete* status. If anything goes wrong while uploading the data, the Job will be aborted instead. Either way, you will receive a response with the details of the Job. *Note that the next screenshot has some blurred information for security reasons. ![Postman upsert (JSON) POST with a customer array body returning the created Job details](../../assets/blog/part-5-data-cloud-mulesoft-integration-5.png) ## Get Job Once you import the Postman collection, you will see a request called "get job info". This is a GET request calling the path `/api/bulk/{id}` of our Mule application. You need to make sure to add your Job ID in the URI of the call and you need to add two Query Parameters, same as you did for the streaming insertion: - `sourceApiName` - `objectName` This request will return the given Job's information (from the ID at the URI). You can use this to track its state. *Note that the next screenshot has some blurred information for security reasons. ![Postman "get job info" GET returning a single Job's JSON with state InProgress](../../assets/blog/part-5-data-cloud-mulesoft-integration-6.png) ## Delete Job Once you import the Postman collection, you will see a request called "delete job". This is a DELETE request calling the path `/api/bulk/{id}` of our Mule application. You only need to make sure to add your Job ID in the URI of the call. This request will delete the Job you pass in the URI. It will only be successful if the given Job is already closed. You can only delete a Job that has one of the following states: - UploadComplete (before it's InProgress, once it's queued you can't delete it) - JobComplete - Aborted - Failed If the Job is in a different state, it will try to Abort it first (this will only be successful if the Job is in an Open state). *Note that the next screenshot has some blurred information for security reasons. ![Postman delete-job DELETE request to the bulk job endpoint returning 200 OK](../../assets/blog/part-5-data-cloud-mulesoft-integration-7.png) And that's all for this post! I hope this helps to understand how to use the bulk operations with our Mule application. Remember that you can always go to my [GitHub repo](https://github.com/alexandramartinez/datacloud-mulesoft-integration) if you want to take a look at the code :) Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## Part 6: CI/CD pipeline with MuleSoft and GitHub Actions - Deploying to CloudHub 2.0 Source: https://prostdev.com/post/part-6-ci-cd-pipeline-with-mulesoft-and-github-actions-deploying-to-cloudhub-2-0 | Published: May 7, 2024 | Category: Tutorials We already learned how to set up secured properties, MUnit coverage, and even a connected app to authenticate when you're using Multi-Factor Authentication (MFA) in your Anypoint Platform account. From now on, we will continue authenticating using a connected app for best practices. What are we missing then? Well, we haven't learned how to enable the CI/CD pipelines for a CloudHub 2.0 deployment. You'd think it's very similar to CloudHub 1.0, but turns out it's very different! Don't worry, I got you :) If you haven’t been following the series or you’re not familiar with GitHub Actions, we recommend you start from the [first article](https://www.prostdev.com/post/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions) to understand how we are setting up all the configurations we need. In this post, we'll learn the basics of setting up our GitHub Actions pipeline to deploy a Mule application to CloudHub 2.0. Once more, using a Connected App. ## Prerequisites - **Anypoint Platform** - You should have an Anypoint Platform account. You can create a new free trial account [here](https://anypoint.mulesoft.com/login/signup). - **GitHub Repo** - You should already have a GitHub repo with a base Mule application and know how to configure secrets in your GitHub repo. If this is not the case, please return to the first article of this series. ## Create a connected app in Anypoint Platform We learned how to create a connected app in the [previous article](https://www.prostdev.com/post/part-5-ci-cd-pipeline-with-mulesoft-and-github-actions-enabling-mfa-through-a-connected-app), but let's go ahead and create a new one. **Sign in to** [Anypoint Platform](https://anypoint.mulesoft.com/) **and navigate to Access Management > Connected Apps.** ![Anypoint Access Management Connected Apps page with the Create app button on the Owned Apps tab.](../../assets/blog/part-6-ci-cd-pipeline-with-mulesoft-and-github-actions-deploying-to-cloudhub-2-0-2.png) **Click on Create app. Add any name you want to identify this app, like github-actions. Select App acts on its own behalf and click on Add Scopes.** > [!DOCS] > You can refer to [this article](https://help.salesforce.com/s/articleView?id=001119708&type=1) to learn more about the minimum scopes for CloudHub 2.0. **Select the following scopes:** - Exchange Contributor - View Environment - View Organization - Create Applications - Delete Applications - Manage Application Data - Manage Runtime Fabrics - Manage Settings - Read Applications Click on Next. Select your Business Group and the Sandbox environment. Review the scopes and click on Add Scopes. You should end up with something like the following. ![Connected app set to act on its own behalf with the nine selected Exchange, General, and Runtime Manager scopes.](../../assets/blog/part-6-ci-cd-pipeline-with-mulesoft-and-github-actions-deploying-to-cloudhub-2-0-3.png) After that, click on Save. Make sure you copy both ID and Secret from the generated connected app. ## Set up your credentials on GitHub Just as we’ve done before, go to your GitHub repository and click on the **Settings** tab. Select **Secrets and variables** > **Actions** and add these two new secrets. - `CONNECTED_APP_CLIENT_ID` - `CONNECTED_APP_CLIENT_SECRET` The values should match what you previously extracted from Anypoint Platform. ## Modify your pom.xml We'll start from the beginning on this so we don't get confused. Starting from a brand new project with no MUnits and no secured properties, open your `pom.xml` file from the root folder. The first thing we'll modify is the groupId. The value here has to match your Business Group ID in Anypoint Platform. Please refer to [this post](https://dev.to/devalexmartinez/4-ways-to-retrieve-your-orgidgroupid-from-anypoint-platform-2e71) to learn how to extract this value. You should end up with something like the following: ```xml 4500658a-7637-4fcf-bc7d-51d1feb28edb ``` Next, change your version so it doesn't say SNAPSHOT. If this is the first version of your app, you can leave it as 1.0.0, like so: ```xml 1.0.0 ``` Then, in the `` tag, change the Mule Maven Plugin version to at least 4.1.0 (we'll be using 4.1.1). ```xml 4.1.1 ``` And here comes the fun part. Inside the mule-maven-plugin, replace the configuration with the following: ```xml org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true mule-application https://anypoint.mulesoft.com MC Sandbox Cloudhub-US-East-2 Repository ${project.artifactId} NONE 1 0.1 true ``` A few notes regarding this configuration: - `` must always be set to MC when deploying to CloudHub 2.0. - `` in this case is Sandbox, but you can change it if you have a different environment set up. - There is only one `` available for your trial account. Before setting this up, you can go to Runtime Manager > Deploy Application and see which region is available for your shared space. In our case, it's US East (Ohio). Refer to [this table](https://docs.mulesoft.com/cloudhub-2/ch2-architecture#regions-and-dns-records) to retrieve the target name you need to set up here. - Whatever you write on `` needs to match in three other places. We'll see exactly where in the rest of the article. In our case, we're setting it up as Repository. - You can use the `` field to choose between NONE, LTS, or EDGE. - NONE will set up Mule Runtime 4.4.0. - LTS will set up the latest available LTS version (changes each year). The last LTS version when this post was written was 4.6.x. This is the most stable option if you want to use 4.6. - EDGE will set up the latest available EDGE version (changes every 4 months). This is a good option if you want to always be on the latest possible version, but a few things might break along the way, so choose carefully. > [!NOTE] > You can set up additional options to customize your deployment even more. For more information about this, see [CloudHub 2.0 Deployment Parameters Reference](https://docs.mulesoft.com/mule-runtime/latest/deploy-to-cloudhub-2#cloudhub2-deploy-reference). Almost done. Now, keep scrolling down the file until you reach the `` tag. Make sure the Anypoint Exchange version is 3 instead of 2. If this is not the case, change it. It should look like the following. ```xml anypoint-exchange-v3 Anypoint Exchange https://maven.anypoint.mulesoft.com/api/v3/maven default ``` Add the following repository. Note that this `` must match the same name you set up before as ``. ```xml Repository Private Exchange repository https://maven.anypoint.mulesoft.com/api/v3/organizations/${project.groupId}/maven default ``` Aaaand finally, make sure you add a `` section at the end (before the `` closing tag) with the following values. Note that this `` must be the same as the previous repository. ```xml Repository Corporate Repository https://maven.anypoint.mulesoft.com/api/v3/organizations/${project.groupId}/maven default ``` Here's the full `pom.xml` for you to confirm everything's good. ```xml 4.0.0 4500658a-7637-4fcf-bc7d-51d1feb28edb test 1.0.0 mule-application test UTF-8 UTF-8 4.1.1 org.apache.maven.plugins maven-clean-plugin 3.1.0 org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true mule-application https://anypoint.mulesoft.com MC Sandbox Cloudhub-US-East-2 Repository ${project.artifactId} NONE 1 0.1 true anypoint-exchange-v3 Anypoint Exchange https://maven.anypoint.mulesoft.com/api/v3/maven default mulesoft-releases MuleSoft Releases Repository https://repository.mulesoft.org/releases/ default Repository Private Exchange repository https://maven.anypoint.mulesoft.com/api/v3/organizations/${project.groupId}/maven default mulesoft-releases mulesoft release repository default https://repository.mulesoft.org/releases/ false org.mule.connectors mule-http-connector 1.9.0 mule-plugin Repository Corporate Repository https://maven.anypoint.mulesoft.com/api/v3/organizations/${project.groupId}/maven default ``` Did you notice we don't have the connected app's client id/secret set up in the `pom.xml`? That is because we will now authenticate using a server in the Maven `settings.xml` file instead of the `pom.xml` file. ## Modify your settings.xml In the previous articles, we've created our own `settings.xml` file in the project to simplify things. Let's do the same here. Create a new .maven folder under the root folder. Then, create a `settings.xml` file inside it. Copy and paste the following into the file. ```xml org.mule.tools Repository ~~~Client~~~ ${client.id}~?~${client.secret} ``` We will be passing the client id/secret through the GitHub Actions file so we're not hardcoding them in the project. > [!NOTE] > If you want to authenticate using username and password instead of connected app, simply add your credentials in this server. Just make sure your user has the necessary permissions. > For more information, see [Authentication Methods](https://docs.mulesoft.com/mule-runtime/latest/deploy-to-cloudhub-2#authentication-methods). ## Modify your build.yml As we've done before, create a .github folder in the root folder of the project, then, create a workflows folder, and a `build.yml` file inside it. Copy and paste the following. ```yaml name: Publish to Exchange & Deploy to CH2.0 on: push: branches: [ main ] jobs: build: runs-on: ubuntu-latest steps: - name: Checkout this repo uses: actions/checkout@v4 - name: Cache dependencies uses: actions/cache@v4 with: path: ~/.m2/repository key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} restore-keys: ${{ runner.os }}-maven- - name: Set up JDK 1.8 uses: actions/setup-java@v4 with: distribution: "zulu" java-version: 8 - name: Publish to Exchange run: | mvn deploy --settings .maven/settings.xml -DskipMunitTests \ -Dclient.id="${{ secrets.CONNECTED_APP_CLIENT_ID }}" \ -Dclient.secret="${{ secrets.CONNECTED_APP_CLIENT_SECRET }}" - name: Deploy to CloudHub 2.0 run: | mvn deploy --settings .maven/settings.xml -DskipMunitTests -DmuleDeploy \ -Dclient.id="${{ secrets.CONNECTED_APP_CLIENT_ID }}" \ -Dclient.secret="${{ secrets.CONNECTED_APP_CLIENT_SECRET }}" ``` As you can see, we are extracting the secrets you set up at the beginning of the article which contain your connected app's client id/secret, and passing them to the Mule runtime as the client.id and client.secret properties. ## Run the pipeline That's it! **Once you’re done with the changes, simply push a new change to the main branch and this will trigger the pipeline.** There is something very important that you need to remember tho. Every time you push a new change, you have to change the version in the `pom.xml`. Otherwise, you will receive an error saying the version has already been published. This happens because the asset is being published to Exchange and then it's deployed to CloudHub and each Exchange asset has to have a different version. ![Successful GitHub Actions build run with steps including Publish to Exchange and Deploy to CloudHub 2.0 all passing.](../../assets/blog/part-6-ci-cd-pipeline-with-mulesoft-and-github-actions-deploying-to-cloudhub-2-0-4.png) Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## New MuleSoft Community Theme for Visual Studio Code! Source: https://prostdev.com/post/new-mulesoft-community-theme-for-visual-studio-code | Published: Apr 30, 2024 | Category: News Hello! I just wanted to take a moment to share this news with you all. I created a theme for Visual Studio Code that is inspired by the MuleSoft colors. Please note that it is NOT an official theme. It's just something I created on my own. I based most colors on Anypoint Platform or other official sites under the [MuleSoft.com](http://MuleSoft.com) domain. However, that doesn't mean they're the official colors :-) I just wanted to mention that. That said...The theme is already published in the VSCode Marketplace. You can download it and start using it right away! > [!BUTTON] > [Download theme](https://marketplace.visualstudio.com/items?itemName=ProstDev.mulesoft-community-theme) ## Contribute You can start contributing via GitHub whenever you see something is missing or *just not right*. You can either create a Pull Request directly in the [GitHub repository](https://github.com/ProstDev/mulesoft-community-theme) or [raise an issue](https://github.com/ProstDev/mulesoft-community-theme/issues) for me to take a look. ## Overview Ok, enough chatting. Let's just see some screenshots! ![Anypoint Code Builder home screen with the Develop an Integration dialog open, in the MuleSoft community theme](../../assets/blog/new-mulesoft-community-theme-for-visual-studio-code-1.png) ![Anypoint Code Builder editing a Mule XML flow with the HTTP Listener config panel and code view, themed in MuleSoft colors](../../assets/blog/new-mulesoft-community-theme-for-visual-studio-code-2.png) ![Anypoint Code Builder Settings page showing Mule runtime and Language Server options in the MuleSoft theme](../../assets/blog/new-mulesoft-community-theme-for-visual-studio-code-3.png) ![Visual Studio Code welcome screen with no folder opened, showing the MuleSoft community theme colors](../../assets/blog/new-mulesoft-community-theme-for-visual-studio-code-4.png) ![Postman running inside VS Code with a Data Cloud SQL query, themed with the MuleSoft palette](../../assets/blog/new-mulesoft-community-theme-for-visual-studio-code-5.png) ![A DataWeave mapping project in VS Code with payload, MyMapping script, and Preview Output panels in the MuleSoft theme](../../assets/blog/new-mulesoft-community-theme-for-visual-studio-code-6.png) ![VS Code command palette listing DataWeave and MuleSoft commands, in the MuleSoft community theme](../../assets/blog/new-mulesoft-community-theme-for-visual-studio-code-7.png) So what do you think?? You can find more information like the color palette and the full list of sections that were customized for this theme in the [Marketplace](https://marketplace.visualstudio.com/items?itemName=ProstDev.mulesoft-community-theme) site. I am always open to feedback. This was my first version of what I envisioned for this theme. And as you can see, I am not a designer! So, a lot of things can be improved for sure :) Don't forget to leave a review in Marketplace so I can improve it. Oh! And this is just the light theme for now. I hope to create a dark theme in the future :D Thank you!! Alex --- ## Interactive tutorial - MuleSoft Anypoint Platform API Catalog by Rolando Carrasco Source: https://prostdev.com/post/interactive-tutorial-mulesoft-anypoint-platform-api-catalog-by-rolando-carrasco | Published: Apr 10, 2024 | Category: Tutorials Have you ever wanted to do some interactive tutorials while learning MuleSoft? There are not a lot of options out there for this purpose. However, MuleSoft Ambassador [Rolando Carrasco](https://www.linkedin.com/in/rolandocarrasco/) took the time to create one for us! Watch the video or follow through the article to learn how to use this awesome platform called [Killercoda](https://killercoda.com/) to learn more about API Catalog. ## Getting started To get started with the interactive tutorial, first go to [this link](https://killercoda.com/borlandc/scenario/mule-tester). This will open the Killercoda platform with the scenario that Rolando created for us. A bunch of stuff is going to be running in the terminal. Let that finish. It'll take a few minutes to get started. ![Killercoda scenario with the intro on the left and packages installing in the terminal on the right](../../assets/blog/interactive-tutorial-mulesoft-anypoint-platform-api-catalog-by-rolando-carrasco-2.png) Once the progress is finished, you will see a new prompt asking you about the sshd_config. Make sure you select "Keep the local version currently installed" and press Enter. ![openssh-server sshd_config prompt with "keep the local version currently installed" highlighted](../../assets/blog/interactive-tutorial-mulesoft-anypoint-platform-api-catalog-by-rolando-carrasco-3.png) It will now continue installing stuff. Once it's 100% finished, you will be able to see the Ubuntu $ prompt on the right screen. At this point, you can click on START (scroll down on the left screen) and get started with the tutorial. ![Killercoda scenario steps with a START button and the Ubuntu terminal prompt ready](../../assets/blog/interactive-tutorial-mulesoft-anypoint-platform-api-catalog-by-rolando-carrasco-4.png) ## Follow the instructions The tutorial's instructions are located to the left side of the screen while the actual terminal is located to the right. Once you start the tutorial, you will see some commands that you need to run in the terminal. Killercoda lets you click on those commands on the left to automatically execute them on the right (on the terminal). You do not need to copy and paste the commands. ![Tutorial step "Install cURL, jq, Node and NPM" with a clickable command block and exec button](../../assets/blog/interactive-tutorial-mulesoft-anypoint-platform-api-catalog-by-rolando-carrasco-5.png) There will also be some instructions for you to follow outside of Killercoda. For example, creating a Connected App in Anypoint Platform or gathering your Organization ID. However, the screenshots in the interactive tutorial are very useful. ![Connected App scopes screenshot in the tutorial instructions, selecting the View Environment scope](../../assets/blog/interactive-tutorial-mulesoft-anypoint-platform-api-catalog-by-rolando-carrasco-6.png) There is also an Editor tab located at the top of the terminal window. ![Killercoda Editor tab open showing a VS Code-style file explorer and the Ubuntu terminal](../../assets/blog/interactive-tutorial-mulesoft-anypoint-platform-api-catalog-by-rolando-carrasco-7.png) This Editor is based on Visual Studio Code and is useful to use the File Explorer to verify certain files have been downloaded to the remote system. You will be using this Editor in the tutorial at some point to modify the Catalog files. ## Other considerations Killercoda lets you use the platform for free but it asks you to finish the tutorial in less than 60 minutes. This is ok. The total time of this tutorial is around 20 minutes. You will have plenty of time to finish. If you have any feedback about this interactive tutorial, make sure to comment on this article or message [Rolando Carrasco](https://www.linkedin.com/in/rolandocarrasco/) directly. We hope to see more interactive tutorials like these from the community! Thanks again Rolando for taking the time and create this amazing tutorial!! Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## Exposing DataWeave: Map+Filter vs. Reduce - which is faster? Source: https://prostdev.com/post/exposing-dataweave-map-filter-vs-reduce-which-is-faster | Published: Mar 26, 2024 | Category: Opinion I was working on some code that was using both map and filter together and I started thinking if there was a better way to refactor this code to make it more performant. Let me first talk about the use case so you get the context of the problem. ## The use case It all starts with a JSON payload that is an array of objects with only three fields (to simplify the example): ```json [ { "id": 1, "criteria": 5, "yearOfBirth": 2000 }, { "id": 2, "criteria": 0, "yearOfBirth": 1990 } ] ``` The goal is to filter out the objects in which the criteria is less than a number (let's say 3 for this example). Plus, some fields are being added to each object. So, the expected output would be something like this: ```json [ { "id": 2, "criteria": 0, "yearOfBirth": 1990, "isValid": true, "additionalField": "something", "years": 34 } ] ``` The object with id 1 was removed because the criteria is not less than 3. I'm sure you have already started thinking of ways to generate this output but we're not there yet. Stay with me. ## Using map and filter The original solution was making use of both map and filter like this: ```dataweave items map { ($), isValid: $.criteria < 3, // needed for filter additionalField: "something", years: now().year - $.yearOfBirth } filter ($.isValid) ``` You might think that I should've done the filter before the map, but that's not the point of this article. Again, stay with me! So I have this code. First I'm doing the map and adding the three new fields, and then I'm doing the filter. The first thing that came to my mind was that I was doing two sets of iterations to the whole array: one for the map and one for the filter. > [!NOTE] > Even if I did the filter before the map, it would've been more than one iteration to the whole array (one full iteration for filter and a partial iteration to map). But again, not the point xD So I started thinking about how I could *reduce* the number of iterations to just one instead of two. And of course, the reduce function came to my mind. ## Using reduce Here we go then. The code I first thought to use was this: ```dataweave items reduce (item, acc=[]) -> if (item.criteria < 3) acc + { (item), isValid: item.criteria < 3, // no longer needed additionalField: "something", years: now().year - item.yearOfBirth } else acc ``` It's pretty much the same thing as the map, but it's only appending the objects in which the criteria is matched and leaving the other objects behind. I figured this was definitely the better approach because now we're only doing one iteration of the whole array and not more than one! ![Smiling Anakin and Padmé Star Wars "for the better, right?" meme top panel](../../assets/blog/exposing-dataweave-map-filter-vs-reduce-which-is-faster-2.png) Right? ![...right?](../../assets/blog/exposing-dataweave-map-filter-vs-reduce-which-is-faster-3.png) ## Timing the approaches I hope you're in disbelief by this point and thinking "*There's no way map+filter is quicker than reduce"* because that's what I thought. So let's see...We can use the [Timer](https://docs.mulesoft.com/dataweave/latest/dw-timer) module to check this out. Especially the [time](https://docs.mulesoft.com/dataweave/latest/dw-timer-functions-time) function. And because I wanted to test these - like, *really* test them - I created some code to generate 10,000 objects. If you want to try it yourself, you can simply up this number in line 4. Also, just because some of you are gonna be wondering about the times of filter+map as opposed to the map+filter, I did that too. Here's the code: > [!PLAYGROUND] > [Open in the DW Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=ProstDev%2Fdataweave-playground-previews&path=scripts%2Ftiming-map-filter-reduce) ```dataweave %dw 2.0 output application/json import time from dw::util::Timer var items = (1 to 10000) as Array map { id: $$, criteria: randomInt(100), yearOfBirth: 1900 + randomInt(123) } fun mapAndFilter() = items map { ($), isValid: $.criteria < 50, // needed for filter additionalField: "something", years: now().year - $.yearOfBirth } filter ($.isValid) fun onlyReduce() = items reduce (item, acc=[]) -> if (item.criteria < 50) acc + { (item), isValid: item.criteria < 50, // can be removed additionalField: "something", years: now().year - item.yearOfBirth } else acc fun filterAndMap() = items filter ($.criteria < 50) map { ($), isValid: $.criteria < 50, // can be removed additionalField: "something", years: now().year - $.yearOfBirth } --- { mapAndFilter: time(() -> mapAndFilter()) then $.end - $.start, onlyReduce: time(() -> onlyReduce()) then $.end - $.start, filterAndMap: time(() -> filterAndMap()) then $.end - $.start } mapObject ((value, key, index) -> (key): (value as String)[2 to -2] as Number ) orderBy $ ``` And here's the result of the previous code (I ran it in the DW Playground) ```json { "filterAndMap": 0.000046, "mapAndFilter": 0.000063, "onlyReduce": 0.124531 } ``` The exact timings will vary every time, but it mostly stays in the same order. We had already predicted that doing filter+map would be faster than map+filter, but had you predicted that reduce would be the slowest?! And by so much more?! If you are skeptical about the results being generated correctly, you can also change the script to the following and run it for every function: ```dataweave time(() -> filterAndMap()) then ( { totalTime: (($.end - $.start) as String)[2 to -2] as Number } ++ $ ) ``` I still see different times depending on the function. Check it out: ![filterAndMap](../../assets/blog/exposing-dataweave-map-filter-vs-reduce-which-is-faster-4.png) ![onlyReduce](../../assets/blog/exposing-dataweave-map-filter-vs-reduce-which-is-faster-5.png) Both generate around 40,000 lines of output. ## Conclusion Nothing is as it seems 🫨 If some milliseconds are a difference for your use case, time your approaches before assuming! Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## Part 4: Data Cloud + MuleSoft integration - Secure your API with basic authentication in API Manager Source: https://prostdev.com/post/part-4-data-cloud-mulesoft-integration | Published: Feb 28, 2024 | Category: Tutorials We already finished our Data Cloud integration with MuleSoft. We have everything deployed and running in CloudHub, and we know how to call this integration to query, insert, or delete records from/to Data Cloud. However, our integration has a fatal flaw: we have no security. In [Part 2](https://www.prostdev.com/post/part-2-data-cloud-mulesoft-integration) of this series, we learned that we should not share the CloudHub URL because everyone had access to our Data Cloud credentials with just the link. There is a simple solution to this problem. In this post, we'll learn how to connect our Mule app (deployed in CloudHub) to API Manager to add basic authentication. This way, even if you share your URL by mistake, people would still need your username/password to access it. ## Prerequisites - **Anypoint Platform** - You should have an Anypoint Platform account. You can create a new free trial account [here](https://anypoint.mulesoft.com/login/signup). It will expire in 30 days but you can create a new one using the same details to register, just making sure to change the username to a different one. - **Deployed Mule app** - Make sure you followed [Part 2](https://www.prostdev.com/post/part-2-data-cloud-mulesoft-integration) of this series before starting. If you have already followed it and deployed version 1.0.0, I'll teach you how to update it to 2.0.0. If there are newer versions, feel free to use those. - **Mule app's JAR** - If you deployed your app with version 1.0.0, make sure you download the version 2.0.0 JAR file. You can find the releases for this Mule project [here](https://github.com/alexandramartinez/datacloud-mulesoft-integration/releases). - **Anypoint Platform's Environment Credentials** - In Anypoint Platform, go to Access Management > Business Groups > select your business group > Environments > Sandbox. Copy the Client ID and Client Secret. See the [docs](https://docs.mulesoft.com/access-management/environments#to-view-the-client-id-and-client-secret-for-an-environment) for more information. - **Postman & the Postman collection** - You can use any other REST Client of your choice, but we used Postman in the [previous article](https://www.prostdev.com/post/part-3-data-cloud-mulesoft-integration), so we'll continue with it. You can download other collections from [this folder](https://github.com/alexandramartinez/datacloud-mulesoft-integration/tree/main/rest-clients-requests). ## Create API in API Manager Sign in to your [Anypoint Platform](https://anypoint.mulesoft.com/) account and head to **API Manager** using the hamburger menu on the top left. ![Anypoint Platform hamburger menu with API Manager highlighted under the APIs section](../../assets/blog/part-4-data-cloud-mulesoft-integration-2.png) In API Manager, click **Add API** > **Add new API**. In the Runtime section, select the **Mule Gateway** runtime and click on **Next**. ![Add API Runtime step with Mule Gateway selected and connect-to-existing-application proxy type](../../assets/blog/part-4-data-cloud-mulesoft-integration-3.png) In the API section, select the **Create new API** option. Add the name **Data Cloud API** and the **HTTP API** asset type. Click on **Next**. ![Add API step creating a new API named Data Cloud API with the HTTP API asset type](../../assets/blog/part-4-data-cloud-mulesoft-integration-4.png) Leave the defaults for Downstream and Upstream (empty) and review the configurations. Click **Save**. ![Review step summarizing the Mule Gateway runtime, Data Cloud API v1, and HTTP downstream scheme](../../assets/blog/part-4-data-cloud-mulesoft-integration-5.png) Copy the **API Instance ID**. We will use it in later steps. ![API Summary page showing the API Instance ID 19307742 and an Unregistered API status](../../assets/blog/part-4-data-cloud-mulesoft-integration-6.png) ## Upgrade the deployed Mule app The previous posts were based on the version 1.0.0 of the JAR file. If you already deployed your Mule app with that version, you'll just have to upload the new 2.0.0 JAR (or newer) to the same application in Runtime Manager. If you deployed your Mule app with this new version already, you can skip this part. > [!TIP] > If you're unsure of the JAR version you deployed, you can check. it by going to your Anypoint Platform account > Runtime Manager > open your app > go to the Settings tab > take a look at the JAR file under **Application File**. It will have the version number. Let's see how to update the JAR from Runtime Manager. Sign in to your Anypoint Platform account and go to **Runtime Manager**. Select the `data-cloud-integration` application you had previously deployed. In the **Settings** tab, locate the **Application File**. Click on **Choose file** > **Upload file** and select the new 2.0.0 JAR (or newer version) you downloaded from the Prerequisites. Don't apply your changes yet. Let's add some more properties first. ![Runtime Manager Settings for the running app, uploading a new JAR via Choose file then Upload file](../../assets/blog/part-4-data-cloud-mulesoft-integration-7.png) ## Add Autodiscovery properties + credentials Before applying your changes, head to the **Properties** tab and switch to the **Text view**. ![Properties tab switched to Text view, with existing protected Salesforce and Data Cloud values below](../../assets/blog/part-4-data-cloud-mulesoft-integration-8.png) Copy the following snippet and replace the placeholders with your values. ```properties anypoint.platform.gatekeeper=flexible api.id=your_api_instance_id anypoint.platform.client_id=your_client_id anypoint.platform.client_secret=your_client_secret ``` > [!NOTE] > If you had already set up the `anypoint.platform.gatekeeper` property to disabled, make sure you change it to flexible. See the [docs](https://docs.mulesoft.com/mule-gateway/mule-gateway-gatekeeper) for more info. ![Properties text view with the gatekeeper, api.id, client_id, and client_secret values filled in](../../assets/blog/part-4-data-cloud-mulesoft-integration-9.png) Go back to the **Table view** and click on **Protect** for your two new Anypoint Platform credentials. Once you're done, click **Apply Changes**. It will take some time to redeploy. ## Add the Basic Authentication policy As you learned from [Part 3](https://www.prostdev.com/post/part-3-data-cloud-mulesoft-integration), you can use Postman to call your integration. If you try sending a request to the Query request now, you should still receive a 200-OK response because we haven't added the security policy yet. ![Postman SELECT query returning a 200 OK with a Data Cloud profile record, before the policy is applied](../../assets/blog/part-4-data-cloud-mulesoft-integration-10.png) Head back to Anypoint Platform > **API Manager**. Select the Data Cloud API we created and make sure the API Status now appears as 🟢 **Active**. You may need to refresh the page. If it doesn't appear as Active yet, make sure your request works and your properties in Runtime Manager are correct. ![API Summary now showing the Data Cloud API status as Active with Mule version 4.4.0](../../assets/blog/part-4-data-cloud-mulesoft-integration-11.png) Then, head to the **Policies** tab on the left and click on **Add policy** under the API-level policies section. ![Policies tab with no policies applied and an Add policy button under API-level policies](../../assets/blog/part-4-data-cloud-mulesoft-integration-12.png) Search for the **Basic Authentication - Simple** policy and click on **Next**. ![Add a policy search for basic authentication with Basic Authentication - Simple selected](../../assets/blog/part-4-data-cloud-mulesoft-integration-13.png) Create your own username/password to access your integration. **Do NOT** use the foo/bar example from the screenshot. Create your own credentials - they can be anything you want. Once you're done, click **Apply**. ![Configure Basic Authentication policy with example User Name foo and User Password bar](../../assets/blog/part-4-data-cloud-mulesoft-integration-14.png) Try calling your Query request once more. After a few seconds, you should receive a **401-Unauthorized** error code. This means the security policy was successfully applied. ![Postman query now returning 401 Unauthorized with a missing-security-context error message](../../assets/blog/part-4-data-cloud-mulesoft-integration-15.png) ## Add your credentials to Postman Now we need to send our new username/password credentials with Postman. There are several ways of doing this. You can: - Hardcode them to each one of the requests - Hardcode them in the collection - Add them to the environment and reference them from each one of the requests - Add them to the environment and reference them from the collection The last one sounds more secure. Let's do that one. In Postman, open. the **Environments** tab and select the **CloudHub** environment we had previously created. Add two new variables: **username** and **password**. Add your values to the Current value field. ![Postman CloudHub environment with host, username, and password variables and their current values](../../assets/blog/part-4-data-cloud-mulesoft-integration-16.png) Click on **Save** and open the **Collections** tab. Click on the Data Cloud Integration collection. This will open a new window with different tabs for the collection. Select the **Authorization** tab and the **Basic Auth** type from the dropdown. Once you're able to see the Username/Password fields with their textboxes, add the reference to your variables with the following syntax: ``` {{variableName}} ``` ![Postman collection Authorization tab set to Basic Auth referencing the username and password variables](../../assets/blog/part-4-data-cloud-mulesoft-integration-17.png) Make sure you **Save** your changes. Now all the requests located inside this collection will inherit the same authentication settings. Try sending the Query request one more time. You should now receive a **200-OK** status code. ![Postman query returning 200 OK again with the Data Cloud profile record after adding credentials](../../assets/blog/part-4-data-cloud-mulesoft-integration-18.png) ## Troubleshooting If you receive a 503 error code when calling the integration from Postman after setting up Autodiscovery (`api.id`), you'll have to wait a few seconds for the security policies to be applied to your API. If a long time has passed and you're still receiving 503 instead of 200, make sure the `api.id` property is correctly set and your API in API Manager appears as Active (make sure to refresh). If you're still experiencing the error, try changing the `anypoint.platform.gatekeeper` property to disabled and try again. Once it works, try changing it back to flexible. ## Conclusion Anypoint Platform offers security policies for your APIs that you can apply using clicks, not code. We used one of the most basic policies for this tutorial, but there are a lot of other policies you can apply to improve your APIs. Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## Part 3: Data Cloud + MuleSoft integration - Call your integration with Postman Source: https://prostdev.com/post/part-3-data-cloud-mulesoft-integration | Published: Feb 20, 2024 | Category: Tutorials A little bird told me that deleting records in Data Cloud is actually not that easy to do. So, I did my research and came up with a Mule application for you all to reuse to (hopefully) make your lives easier when dealing with Data Cloud! In this third part, we'll learn how to use our integration. We'll use [Postman](https://www.postman.com/) for this article, but you can use any other REST client like [Thunder Client](https://www.thunderclient.com/), [cURL](https://curl.se/), or [Advanced REST Client](https://www.advancedrestclient.com/). ## Prerequisites - **Postman collection** - Download the Postman collection from [this link](https://github.com/alexandramartinez/datacloud-mulesoft-integration/tree/main/rest-clients-requests/postman). You can also find other collections/requests [here](https://github.com/alexandramartinez/datacloud-mulesoft-integration/tree/main/rest-clients-requests). - **Postman** - Make sure you download [Postman](https://www.postman.com/). You can also use the web version by creating a Postman account, instead of installing the app locally. - **Mule app URL** - By the end of the [previous article](https://www.prostdev.com/post/part-2-data-cloud-mulesoft-integration), we retrieved the public Endpoint/URL of our deployed Mule application. Make sure you have this before starting. ## Import the collection In Postman, make sure you're located inside the **Collections** tab and click the **Import** button. ![Postman Collections tab with the Import button highlighted](../../assets/blog/part-3-data-cloud-mulesoft-integration-2.png) Select the Postman collection you downloaded from the Prerequisites. After you import it, you should now have something like the following. > [!NOTE] > The following screenshot contains only the **Streaming** insert/delete operations (from when the post was created). The **Bulk** operations are introduced after the JAR version 2.1.0, updated on July 2024. ![Imported Data Cloud Integration collection with schema, query, insert and delete requests](../../assets/blog/part-3-data-cloud-mulesoft-integration-3.png) ## Set up the environment Select the **Environments** tab from the left (next to Collections). We could create a Global variable, but we're going to create a new environment to follow best practices. Click on **Create Environment**. ![Empty Postman Environments tab with a Create Environment link](../../assets/blog/part-3-data-cloud-mulesoft-integration-4.png) Let's name this environment **CloudHub**. Add a variable called **host** and the current value field will be your Mule app URL. ![CloudHub Postman environment with a host variable set to the Mule app URL](../../assets/blog/part-3-data-cloud-mulesoft-integration-5.png) Save the variable by clicking on the **Save** button at the top-right. Select this new environment from the environment dropdown located at the top-right (over the Save button). ![Postman environment dropdown listing No Environment and the CloudHub environment](../../assets/blog/part-3-data-cloud-mulesoft-integration-6.png) Go back to the **Collections** tab. You can now run the requests! ## schema With this endpoint, you can send a JSON object to be transformed into the OpenAPI YAML schema. This is needed to create your Ingestion API and Data Stream in Data Cloud. Because the Ingestion API doesn't accept nested objects on the schema, this endpoint will transform your multi-level object into the single level needed for Data Cloud. For example, if you paste the following input payload under the **Body** tab of the request: ```json { "customer": { "id": 1, "first_name": "Alex", "last_name": "Martinez", "email": "alex@sf.com", "address": { "street": "415 Mission Street", "city": "San Francisco", "state": "CA", "postalCode": "94105", "geo": { "lat": 37.78916, "lng": -122.39521 } } } } ``` ![Postman schema POST request with a nested customer JSON object in the raw body](../../assets/blog/part-3-data-cloud-mulesoft-integration-7.png) it would be first flattened and transformed into the following output: ```json { "customer": { "id": 1, "first_name": "Alex", "last_name": "Martinez", "email": "alex@sf.com", "street": "415 Mission Street", "city": "San Francisco", "state": "CA", "postalCode": "94105", "lat": 37.78916, "lng": -122.39521 } } ``` then, based on this new input, you will receive the YAML schema like the following: ```yaml openapi: 3.0.3 components: schemas: customer: type: object properties: id: type: number first_name: type: string last_name: type: string email: type: string street: type: string city: type: string state: type: string postalCode: type: string lat: type: number lng: type: number ``` If you want to change the OpenAPI version, you will have to set a different version using the openapiversion query parameter. To do this, go to the **Params** tab inside the request and modify the value. ![Postman Params tab showing the openapiversion query parameter](../../assets/blog/part-3-data-cloud-mulesoft-integration-8.png) > [!NOTE] > This is the only request that does not connect to Data Cloud and is not using your Data Cloud credentials. If you need to use this request before setting up your Salesforce configurations, you can go through the Mule app deployment (Part 2) and just input random credentials as a placeholder for the secured properties. You can modify them later in CloudHub. ## query Send your SOQL query in the body of the request in a text/plain format. You can modify the query by going to the **Body** tab inside the request. ![Postman query POST request with a SOQL SELECT statement in the body](../../assets/blog/part-3-data-cloud-mulesoft-integration-9.png) You will receive a JSON response with the result from Data Cloud. For example, from the previous query, you'd receive a JSON Array with the results of the SELECT: ```json [ { "DataSourceObject__c": "MuleSoft_Ingestion_API_runner_profiles_38447E8E", "DataSource__c": "MuleSoft_Ingestion_API_996db928_2078_4e3a_9c67_1c80b32790aa", "city__c": "Toronto", "created__c": "2017-07-21", "email__c": "alex@sf.com", "first_name__c": "Alex", "gender__c": "NB", "last_name__c": "Martinez", "maid__c": 1.000000000000000000, "state__c": "ON" } ] ``` If there are no records matching the query, you'll receive an empty array ([ ]). ## insert Make sure you add the following query parameters (in the **Params** tab) to let Data Cloud know more information about where you want to insert new records: ![Postman insert request Params tab with sourceApiName and objectName query parameters](../../assets/blog/part-3-data-cloud-mulesoft-integration-10.png) Next, in the body of the request (**Body** tab), make sure to use a JSON Array. Each Object inside this Array is a new record. For example: ```json [ { "maid": 1, "first_name": "Alex", "last_name": "Martinez", "email": "alex@sf.com", "gender": "NB", "city": "Toronto", "state": "ON", "created": "2017-07-21" } ] ``` ![Postman insert request body with a JSON array of records to add](../../assets/blog/part-3-data-cloud-mulesoft-integration-11.png) > [!IMPORTANT] > Streaming in Data Cloud in limited to max 200 records per insertion. If everything runs smoothly, you will receive a **200 - OK** successful response. > [!NOTE] > It may take a few minutes for your data to be updated in Data Cloud. You can manually check the records in Data Cloud or wait to attempt the `/query` from your MuleSoft API. ## delete Make sure you add the following query parameters (in the **Params** tab) to let Data Cloud know more information about where you want to insert new records: ![Postman delete request Params tab with sourceApiName and objectName query parameters](../../assets/blog/part-3-data-cloud-mulesoft-integration-12.png) Next, in the body of the request, make sure to use a JSON Array. Each Object inside this Array is the Primary Key of the record to delete. For example: ```json [ 1 ] ``` ![Postman delete request body with a JSON array of primary keys to remove](../../assets/blog/part-3-data-cloud-mulesoft-integration-13.png) > [!IMPORTANT] > Streaming in Data Cloud in limited to max 200 records per deletion. If everything runs smoothly, you will receive a **200 - OK** successful response. > [!NOTE] > It may take a few minutes for your data to be updated in Data Cloud. You can manually check the records in Data Cloud or wait to attempt the /query from your MuleSoft API. ## Conclusion That's it! That's all you need to know to get your Mule application going. I will be working on more enhancements to this app and I'll be writing more articles for you to learn how to implement/edit this code on your own. Keep posted! Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## Part 2: Data Cloud + MuleSoft integration - Deploy your own Mule app on Anypoint Platform (CloudHub) Source: https://prostdev.com/post/part-2-data-cloud-mulesoft-integration | Published: Feb 13, 2024 | Category: Tutorials A little bird told me that deleting records in Data Cloud is actually not that easy to do. So, I did my research and came up with a Mule application for you all to reuse to (hopefully) make your lives easier when dealing with Data Cloud! In this second part, we'll go through the MuleSoft side of the integration and you'll deploy your own Mule app to CloudHub. You do not have to know MuleSoft beforehand. I will guide you through every step. ## Prerequisites - **Anypoint Platform** - You should have an Anypoint Platform account. You can create a new free trial account [here](https://anypoint.mulesoft.com/login/signup). It will expire in 30 days but you can create a new one using the same details to register, just making sure to change the username to a different one. - **Mule app's JAR** - You will need to download the Mule application in a JAR format. You can find the releases for this Mule project [here](https://github.com/alexandramartinez/datacloud-mulesoft-integration/releases). By the time this post was created, the latest release was v1.0.0. However, if there is new functionality added later, it might be better for you to use the latest version. - **Salesforce credentials** - Make sure you followed the [previous post](https://www.prostdev.com/post/part-1-data-cloud-mulesoft-integration) because you will need some of those credentials to follow this post. You will need: - `salesforce.username` - The username you use to log in to [Salesforce](https://login.salesforce.com/) - `salesforce.password` - The password you use to log in to [Salesforce](https://login.salesforce.com/) - `cdp.consumer.key` - The Consumer Key for the Connected App you created in Salesforce - `cdp.consumer.secret` - The Consumer Secret for the Connected App you created in Salesforce ## Deploy application Sign in to your [Anypoint Platform](https://anypoint.mulesoft.com/) account and navigate to Runtime Manager from the left-hand side menu. ![Anypoint Platform header with the hamburger Products menu button](../../assets/blog/part-2-data-cloud-mulesoft-integration-2.png) ![Anypoint Platform products menu with Runtime Manager highlighted](../../assets/blog/part-2-data-cloud-mulesoft-integration-3.png) If this is your first time opening Runtime Manager, you will be asked to choose an environment. Choose **Sandbox**. Once inside, click on the blue **Deploy application** button. ![Empty Runtime Manager Applications view in the Sandbox with a Deploy application button](../../assets/blog/part-2-data-cloud-mulesoft-integration-4.png) Add any application name—for example, `data-cloud-integration`. **CloudHub 2.0** should be selected by default under Deployment Target, if not, please select it. Under Application File, select **Choose file** > **Upload File** and select the JAR file you downloaded from the Prerequisites. ![Deploy Application form with app name, CloudHub 2.0 target and the uploaded JAR file](../../assets/blog/part-2-data-cloud-mulesoft-integration-5.png) > [!NOTE] > We will deploy to CloudHub 2.0, but if you're deploying to CloudHub 1.0, the application name has to be unique across all applications worldwide. If the application name is not available, you can add your username at the end to make it unique. For example: `data-cloud-integration-amartinez`. ## Runtime tab Under the **Runtime** tab (which should be already open), make sure the Release Channel is set to **None** so we can select the Runtime Version **4.4.0**. You can experiment with different runtimes if you want! However, when this app was created and this post was published, I used version 4.4.0. I'll document in future versions of the JAR file if a new runtime is supported. > [!NOTE] > If you use the JAR version 2.1.0 or newer, you can select the runtime 4.7.0 and/or Java 11/17. You can leave the rest of the settings with the default values. ![Runtime tab with Release Channel set to None and Runtime Version 4.4.0](../../assets/blog/part-2-data-cloud-mulesoft-integration-6.png) ## Properties tab Now go to the **Properties** tab. Click on the **Text view** switch so you can copy and paste the following properties (add your actual credentials for each property): ```properties salesforce.username=your_username salesforce.password=your_password cdp.consumer.key=your_key cdp.consumer.secret=your_secret anypoint.platform.gatekeeper=disabled ``` > [!NOTE] > The last property, `anypoint.platform.gatekeeper` has to be added for the JAR version 2.0.0 and newer. You can skip it if you're using the 1.0.0 version. ![Properties tab Text view with the Salesforce and Data Cloud credentials pasted in](../../assets/blog/part-2-data-cloud-mulesoft-integration-7.png) Switch back to the **Table view** and click on the **Protect** button next to each of the values (this button is only available for CloudHub 2.0). ![Properties tab Table view with a Protect button next to each property value](../../assets/blog/part-2-data-cloud-mulesoft-integration-8.png) Now your credentials are not visible for added security. After this, click on **Deploy Application**. There is no need to modify the Ingress or Logging tabs. ![Properties table showing each credential value as a protected value](../../assets/blog/part-2-data-cloud-mulesoft-integration-9.png) ## Copy the URL Wait a few seconds for the JAR file to be uploaded. After this, you might see your app has a 🔴 red circle. Don't worry about this. If you go to the **Settings** tab you will see the yellow circle saying the configuration is still being applied. Wait a few more minutes until it's deployed 🟢 (green circle). It can take up to 10 minutes. ![App Settings showing data-cloud-integration applying configuration with status Not running](../../assets/blog/part-2-data-cloud-mulesoft-integration-10.png) Once the app has been deployed and the Application Status says 🟢 Running, you can retrieve the **public endpoint**/URL from either the **Settings** or the **Dashboard** tab. ![App Settings with status Running and the public endpoint CloudHub URL shown](../../assets/blog/part-2-data-cloud-mulesoft-integration-11.png) Copy this URL. We will use it in the next article to call the integration. > [!CAUTION] > **DO NOT** share this URL publicly. Anyone with this URL will be able to access your Data Cloud API (my URL from the previous screenshot is no longer available). > > If you wish to create a new URL, you will have to delete this app from the Settings tab (where it says Stop at the top-right) and deploy a new one with a new name following the same steps. > > You can **Stop** and **Start** your app only when you are using it to avoid unwanted requests. ## Conclusion After you follow all these steps, you should now have the public endpoint/URL that we will use to call our Mule application. The Mule app will then connect with Data Cloud using the credentials we provided in the settings. Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## Part 1: Data Cloud + MuleSoft integration - Connected App, Ingestion API & Data Stream settings in Salesforce Source: https://prostdev.com/post/part-1-data-cloud-mulesoft-integration | Published: Feb 7, 2024 | Category: Tutorials A little bird told me that deleting records in Data Cloud is actually not that easy to do. So, I did my research and came up with a Mule application for you all to reuse to (hopefully) make your lives easier when dealing with Data Cloud! In this first part, we'll go through the Salesforce/Data Cloud settings that we need to set up before even calling Data Cloud through the Mule app. > *I want to thank* [Mohammad Arsha](https://www.linkedin.com/in/mohammad-arsha-7b18381b6/) *for creating this useful* [article](https://medium.com/another-integration-blog/connect-data-cloud-with-mulesoft-bfad40fe2e8e) *that got me started on the Data Cloud/Salesforce configuration. Keep creating content!* ## Prerequisites - **Salesforce Account with Data Cloud** - You should have a Salesforce account where you will be able to access Data Cloud. For more info, see [Salesforce Data Cloud](https://www.salesforce.com/products/data/). ## Connected App settings The first thing we're going to do is to create a Connected App in [Salesforce](https://login.salesforce.com/) to authenticate to Data Cloud. To do this, first enter the **Setup** from the top-right of the screen. ![Salesforce gear menu showing the Setup option for the current app](../../assets/blog/part-1-data-cloud-mulesoft-integration-2.png) Search for **App Manager** using the Quick Find input from the left and click on **New Connected App** on the top-right of the screen. ![Salesforce App Manager with the New Connected App button highlighted](../../assets/blog/part-1-data-cloud-mulesoft-integration-3.png) Fill in the following fields and click **Save** and **Continue** once you're done. | **Field** | **Value** | |---|---| | Connected App Name | MuleSoft Integration Connected App | | API Name | `MuleSoft_Integration_Connected_App` | | Contact Email | your email | | Enable OAuth Settings | ✅ | | Callback URL | [https://login.salesforce.com/services/oauth2/callback](https://login.salesforce.com/services/oauth2/callback) | | Selected OAuth Scopes | - Access Interaction API resources (`cdp_api`)
- Access all Data Cloud API resources (`cdp_api`)
- Manage Data Cloud Calculated Insight data (`cdp_calculated_insight_api`)
- Manage Data Cloud Identity Resolution (`cdp_identityresolution_api`)
- Manage Data Cloud Ingestion API data (`cdp_ingest_api`)
- Manage Data Cloud profile data (`cdp_profile_api`)
- Manage user data via Web browsers (`web`)
- Perform ANSI SQL queries on Data Cloud data (`cdp_query_api`)
- Perform segmentation on Data Cloud data (`cdp_segment_api`) | | Enable Client Credentials Flow | ✅ | In the created Connected App detail page, click on **Manage Consumer Details** ![Connected App API section with the Manage Consumer Details button](../../assets/blog/part-1-data-cloud-mulesoft-integration-4.png) Enter the code sent to your email and click on **Verify**. Once inside the Consumer Details page, click on **Generate** and copy the newly generated **Staged Consumer Key** and **Staged Consumer Secret**. You will use these credentials for your Mule integration. After that, click on **Apply** and **Apply** again. Your staged consumer details should be now your main consumer details. ![Connected App Consumer Details page with masked Consumer Key and Secret and Copy buttons](../../assets/blog/part-1-data-cloud-mulesoft-integration-5.jpg) > [!IMPORTANT] > These credentials will be used as `cdp.consumer.key` and `cdp.consumer.secret` in your Mule application's settings in Anypoint Platform (the properties in Runtime Manager). You can close this window after you've copied the credentials. ## OAuth settings Still inside the **Setup** page, search for **OAuth and OpenID Connect Settings** in the Quick Find input box and click on it. Make sure the **Allow OAuth Username-Password Flows** option is **On**. ![OAuth and OpenID Connect Settings with Allow OAuth Username-Password Flows toggled On](../../assets/blog/part-1-data-cloud-mulesoft-integration-6.png) ## Ingestion API settings If you already have an Ingestion API **with a YAML schema**, just take note of the name of the Ingestion API you created. Otherwise, follow the next steps to create one. Still inside the **Setup** page, search for **Ingestion API** in the Quick Find input box and click on it. Click on **New**. ![Data Cloud Ingestion API setup page with the New button to create a connector](../../assets/blog/part-1-data-cloud-mulesoft-integration-7.png) Add a Connector Name like **MuleSoft Ingestion API** and click on **Save**. ![MuleSoft Ingestion API connector details showing Source API Name MuleSoft_Ingestion_API](../../assets/blog/part-1-data-cloud-mulesoft-integration-8.png) > [!IMPORTANT] > Take note of the **Source API Name**. This name will be used as the `sourceApiName` query parameter to call the Mule application. In this case, the correct value would be `MuleSoft_Ingestion_API`. If you already have a YAML schema, upload it here and take note of the name of the object(s). If not, upload the following example schema (the names of the objects are `runner_profiles` and `exercises`). ```yaml openapi: 3.0.3 components: schemas: runner_profiles: type: object properties: maid: type: number first_name: type: string last_name: type: string email: type: string gender: type: string city: type: string state: type: string created: type: string format: date exercises: type: object properties: runid: type: number datetime: type: string format: date-time km_run: type: number calories_burned: type: number duration_minutes: type: number maid: type: number type: type: string ``` ![Ingestion API schema listing the exercises and runner_profiles objects, connector In Use](../../assets/blog/part-1-data-cloud-mulesoft-integration-9.png) Click on **Save**. > [!IMPORTANT] > Whichever object you wish to insert/delete records to, will be used as the `objectName` query parameter to call the Mule application. In this case, the correct value(s) would be `runner_profiles` or `exercises`. ## Data Stream settings Exit the Setup page or go back to the main page after logging in to Salesforce. Click on the apps button at the top-left of the screen and search for **Data Cloud**. Click on the app. ![Salesforce App Launcher searching for and showing the Data Cloud app](../../assets/blog/part-1-data-cloud-mulesoft-integration-10.png) Select the **Data Streams** tab from the top. Click **New**. ![Data Cloud Data Streams tab with the New button to create a data stream](../../assets/blog/part-1-data-cloud-mulesoft-integration-11.png) Select **Ingestion API** and click **Next**. ![New Data Stream source picker with the Ingestion API tile selected](../../assets/blog/part-1-data-cloud-mulesoft-integration-12.png) Select your **MuleSoft Ingestion API** from the dropdown. Select all the objects and click on **Next**. - In the `exercises` configuration, select Category **Profile** and Primary Key `maid` - In the `runner_profiles` configuration, select Category **Profile** and Primary Key `maid` Click **Next**. Select the **default** Data Space and click **Deploy**. ![Two deployed data streams: MuleSoft Ingestion API-runner_profiles and -exercises](../../assets/blog/part-1-data-cloud-mulesoft-integration-13.png) If you used the example YAML schema, you should have ended up with something like the following. ![runner_profiles data stream fields table with maid set as the Primary Key](../../assets/blog/part-1-data-cloud-mulesoft-integration-14.png) > [!NOTE] > In this Data Stream, we selected `maid` as the Primary Key. This is the field we're going to need per record to delete them using the Mule app. ## Conclusion After finishing this configuration in Salesforce/Data Cloud, you should now have the following data that you will use later when calling the integration. | **Field** | **Value** | |---|---| | `salesforce.username` | The username you used to log in to Salesforce at [login.salesforce.com](http://login.salesforce.com) | | `salesforce.password` | The password you used to log in to Salesforce at [login.salesforce.com](http://login.salesforce.com) | | `cdp.consumer.key` | The Consumer Key you generated from the Connected App | | `cdp.consumer.secret` | The Consumer Key you generated from the Connected App | | `sourceApiName` | The Source API Name from the Ingestion API (`MuleSoft_Ingestion_API`) | | `objectName` | The objects created from the YAML schema in the Ingestion API (`runner_profiles` or `exercises`) | | Object ID / Primary Key | The field selected as Primary Key from the Data Stream settings (`maid`) | Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## How to add JVM/Command-line arguments to the Mule 4 Runtime in Anypoint Code Builder (ACB) Source: https://prostdev.com/post/how-to-add-jvm-command-line-arguments-to-the-mule-4-runtime-in-anypoint-code-builder-acb | Published: Jan 25, 2024 | Category: Tutorials I've received this question a lot lately, so I figured I'd create a quick post about it. Adding JVM or Command-line arguments to your Mule Runtime is especially useful when you're using properties files that are separated by environment. It is also useful to pass a secure/encryption key to your Mule application when you have secured properties. Let's learn how to pass these arguments in ACB. ## Prerequisites - **Visual Studio Code** - To use Anypoint Code Builder (Desktop), you must first install VS Code. You can download it [here](https://code.visualstudio.com/). - **Anypoint Extension Pack** - To enable Anypoint Code Builder in your Visual Studio Code, you must install this extension. You can install it from [here](https://marketplace.visualstudio.com/items?itemName=salesforce.mule-dx-extension-pack). - **REST Client** - A lot of people use [Postman](https://www.postman.com/). You can also use [curl](https://curl.se/) if you're more into the command line, or [Thunder Client for VS Code](https://marketplace.visualstudio.com/items?itemName=rangav.vscode-thunder-client). We'll use Thunder Client for this post. - **GitHub Repo** - If you want to follow along with the code I generated, you can check it out [here](https://github.com/ProstDev/args-in-acb). ## Create the Mule project You can use the GitHub repository listed in the prerequisites, or you can generate your code by following the next instructions. In ACB, click on **Develop an Integration** and give a name to the project. In our case, it will be test-project. Click on **Create project**. Paste the following code before the `` tag. ```xml ``` This will create an HTTP Listener and a Set Payload component in the flow. You will also create two Global Element Configurations: one for the HTTP Listener configuration and another for the properties, in this case, using the ${env}.properties syntax. > [!NOTE] > You can use .properties or .yaml extensions for your properties files. Create a new `dev.properties` file under `src/main/resources` and paste the following properties: ``` http.listener.host=0.0.0.0 http.listener.port=8081 ``` > [!NOTE] > You should have a different properties file per environment. For now, we are only creating the basic configuration for the dev environment to demonstrate. ## Add arguments to the runtime Once you have the basic setup to run the Mule app, we'll now add the env argument to the list. So, every time you run this Mule app, the env property will be provisioned to the runtime. Open the ACB Settings by clicking on the ⚙️ icon on the home screen. ![Anypoint Code Builder home screen with the gear icon tooltip reading "Open ACB Settings".](../../assets/blog/how-to-add-jvm-command-line-arguments-to-the-mule-4-runtime-in-anypoint-code-builder-acb-2.png) Go to **Mule > Runtime: Default Arguments** and append the following argument at the end of the value. ``` -M-Denv=dev ``` ![VS Code settings showing Mule Runtime Default Arguments with -M-Denv=dev appended to the value.](../../assets/blog/how-to-add-jvm-command-line-arguments-to-the-mule-4-runtime-in-anypoint-code-builder-acb-3.png) ## Run your app locally You can start your Mule app from the **Run and Debug** tab and click on the green run button. ![VS Code Run and Debug tab with the green "Debug Mule" run button to start the app.](../../assets/blog/how-to-add-jvm-command-line-arguments-to-the-mule-4-runtime-in-anypoint-code-builder-acb-4.png) Finally, call your Mule application from your REST Client with the following URL: ``` localhost:8081/hello ``` You should receive a "Hello World" response. ![Thunder Client GET to localhost:8081/hello returning 200 OK with a "Hello World" response.](../../assets/blog/how-to-add-jvm-command-line-arguments-to-the-mule-4-runtime-in-anypoint-code-builder-acb-5.png) Ta-da!! I hope that was helpful. Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## How to RE-scaffold Mule flows from an Exchange API specification in Anypoint Studio Source: https://prostdev.com/post/re-scaffold-mule-flows-exchange-api-spec-studio | Published: Dec 12, 2023 | Category: Tutorials What happens when there is a change to a published API in Anypoint Exchange? How can we get these changes re-scaffolded to our API implementation once we have already done the scaffolding? In this post, we'll learn how to re-scaffold our Mule project based on new changes done to the published API Specification in Exchange. ## Prerequisites - **Anypoint Platform** - You should have an Anypoint Platform account. You can create a new free trial account [here](https://anypoint.mulesoft.com/login/signup). - **API specification** - You should already have an API specification published in Anypoint Exchange. - **Anypoint Studio** - MuleSoft's IDE based on Eclipse. You can download it [here](https://www.mulesoft.com/lp/dl/anypoint-mule-studio). - **Scaffolded Mule project** - You should already have scaffolded the first version of the Mule project based on a previous version of the API specification since we're going to re-scaffold the same project with the new change in the specification. If you haven't done this yet, please refer to [part 1](https://www.prostdev.com/post/scaffold-mule-flows-from-published-api-specification) of this series. - **GitHub Repo** - If you want to follow along with the code I generated, you can check it out [here](https://github.com/ProstDev/codetober23/tree/main/day3). ## Update the API specification's dependency version Once our API Specification has been updated and published to Anypoint Exchange, our Mule project will become outdated and this is where we'll do the re-scaffolding to update the project to match the latest API specification version from Exchange. In your Package Explorer in Anypoint Studio (located at the left of the screen), you will be able to find the name of your API Specification with the first version (in this case v1.0.0). If you open the dependency, you'll be able to see the .zip associated with the specification. ![Studio Package Explorer showing the my-to-do-api v1.0.0 dependency and its RAML zip](../../assets/blog/re-scaffold-mule-flows-exchange-api-spec-studio-2.png) We want to change this version of the specification to the next available version, which in our case is v2.0.0, but it could be v1.0.1 or any other version your specification has as the latest. To do this, right-click on your API specification's dependency and select the name of the spec (in this case **my-to-do-api**). After that, select **Update Version**. ![Right-click menu on the API dependency with the Update Version option highlighted](../../assets/blog/re-scaffold-mule-flows-exchange-api-spec-studio-3.png) This will open a new window where you'll be able to refresh the APIs associated with the Mule Project by clicking the **Refresh** or **Update** button on the top-right side. ![Mule Project APIs settings listing my-to-do-api at version 1.0.0 with a refresh button](../../assets/blog/re-scaffold-mule-flows-exchange-api-spec-studio-4.png) > [!IMPORTANT] > If nothing happens when you click this button, it might mean that your credentials have expired. You will need to refresh them from **Anypoint Studio > Settings > Anypoint Studio > Authentication**. Remove your current Anypoint Platform credentials from this window and re-add them. Try refreshing the APIs one more time after refreshing the credentials. Once the APIs are refreshed, you'll be able to see that there is a newer version of the API specification you're using in your project. ![Tooltip on the API version 1.0.0 reading "There is a newer version: 2.0.0"](../../assets/blog/re-scaffold-mule-flows-exchange-api-spec-studio-5.png) Click on the **Update version** button for this API to be refreshed. ![API dependency detail showing "New version available" with an Update version button](../../assets/blog/re-scaffold-mule-flows-exchange-api-spec-studio-6.png) Once it's refreshed, it will appear with the newest version. In this case, 2.0.0. Click **Apply and Close**. ![Properties APIs view with My To-Do API now at version 2.0.0 and an Apply button](../../assets/blog/re-scaffold-mule-flows-exchange-api-spec-studio-7.png) Immediately after that, you will get a message asking if you want to scaffold the API specification. Select **Yes** to re-scaffold it. ![Dialog asking "Do you want to scaffold 'My To-Do API' specification?" with Yes and No](../../assets/blog/re-scaffold-mule-flows-exchange-api-spec-studio-8.png) After that, your API specification's dependency will appear with the new version in the Package Explorer and you'll be able to see your new changes in the API implementation. ![Package Explorer showing the My To-Do API dependency updated to v2.0.0](../../assets/blog/re-scaffold-mule-flows-exchange-api-spec-studio-9.png) Subscribe to receive notifications as soon as new content is published ✨ 💬 Prost! 🍻 --- ## How to scaffold Mule flows from a published API spec in Anypoint Code Builder (ACB) Source: https://prostdev.com/post/scaffold-mule-flows-from-published-api-spec-acb | Published: Dec 7, 2023 | Category: Tutorials In this post, I am going to show you how to scaffold your flows in Anypoint Code Builder from a published API specification. > [!DOCS] > To do this same thing in Anypoint Studio, refer to [How to scaffold Mule flows from a published API spec in Anypoint Studio](https://www.prostdev.com/post/scaffold-mule-flows-from-published-api-specification) Why would you want to scaffold Mule flows from an API specification? This way you will be able to get started on your Mule application with a base project that is created upon your specification, instead of starting the Mule project and Mule flows from scratch. Once you scaffold the Mule project, you will have: - Mule flows for each HTTP method in your specification - Basic error handling for different HTTP status codes - Initial Mule project with some Transform Message components where applicable ## Prerequisites - **Anypoint Platform** - You should have an Anypoint Platform account. You can create a new free trial account [here](https://anypoint.mulesoft.com/login/signup). - **API specification** - You should already have an API specification published in Anypoint Exchange. - **Visual Studio Code** - To use Anypoint Code Builder, you must first install VS Code. You can download it [here](https://code.visualstudio.com/). - **Anypoint Extension Pack** - To enable Anypoint Code Builder in your Visual Studio Code, you must install this extension. You can install it from [here](https://marketplace.visualstudio.com/items?itemName=salesforce.mule-dx-extension-pack). - **GitHub Repo** - If you want to follow along with the code I generated, you can check it out [here](https://github.com/ProstDev/codetober23/tree/main/day2). > [!DOCS] > If this is your first time creating an API specification, refer to [How to use MuleSoft's visual API Designer to create a To-Do API specification using clicks, not code](https://www.prostdev.com/post/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code) ## Create a Mule project in ACB Once you are inside Anypoint Code Builder, select the **Implement an API** quick action. ![Anypoint Code Builder Quick Actions panel with the Implement an API option highlighted](../../assets/blog/scaffold-mule-flows-from-published-api-spec-acb-2.png) This will open a new menu or window, and it will ask you to sign in to Anypoint Platform to retrieve the API Specification from your account. Click **Allow** and follow the prompts to sign in. Once you are signed in, you will be able to search an API Specification from Exchange to implement it. Simply search for your asset's name on the search bar and select it from the list by clicking on **Add Asset**. ![New project form with name todoapp and an Exchange search listing the My To-Do API with Add Asset](../../assets/blog/scaffold-mule-flows-from-published-api-spec-acb-3.png) > [!NOTE] > If you're having issues searching for your asset, you can use the **Show filters** toggle to search for more specific assets. For example, using your own organization or the asset's type. After the API Specification has been selected, click on **Create Project**. Wait a few seconds for everything to finish processing and you'll have the finalized project after that. ## Navigate the generated project Once the project has been created, you should be able to see the flows in the configuration file located under `src/main/mule`. ![Scaffolded todoapp.xml shown as a flow canvas alongside its generated Mule XML](../../assets/blog/scaffold-mule-flows-from-published-api-spec-acb-4.png) You can click on the **Flow List** button from the top-left of the graphical view (or the flow canvas) to see a complete list of all the flows that have been implemented in the current XML file. ![Flow List panel listing the generated GET, POST, PUT, and DELETE flows for the to-do API](../../assets/blog/scaffold-mule-flows-from-published-api-spec-acb-5.png) The Error Handling has been added automatically to the main flow as well. This contains the HTTP status codes for some basic errors like 400 Bad Request or 404 Not Found. ![Auto-generated error handling with On Error Propagate blocks each routing to a Transform Message](../../assets/blog/scaffold-mule-flows-from-published-api-spec-acb-6.png) Notice how some of the flows already have a Transform Message component for you to see which DataWeave code is needed to retrieve some data like the URI Parameter(s). This, of course, is just in the case that your API Specification contains a URI Parameter. ![Transform Message in a GET flow with DataWeave setting a variable from attributes.uriParams.id](../../assets/blog/scaffold-mule-flows-from-published-api-spec-acb-7.png) There will also be Transform Message components added for some cases where you have set up an example in the API Specification. ![Transform Message with DataWeave returning a sample JSON to-do payload built from the spec example](../../assets/blog/scaffold-mule-flows-from-published-api-spec-acb-8.png) Pretty cool, right? Keep posted for some more articles on scaffolding in Anypoint Studio or Anypoint Code Builder! Remember to subscribe at the bottom of the page to receive email notifications as soon as new content is published. ✨ 💬 Prost! 🍻 --- ## How to scaffold Mule flows from a published API spec in Anypoint Studio Source: https://prostdev.com/post/scaffold-mule-flows-from-published-api-specification | Published: Nov 29, 2023 | Category: Tutorials In this post, I am going to show you how to scaffold your flows in Anypoint Studio from a published API specification. > [!DOCS] > To do this same thing in Anypoint Code Builder, refer to [How to scaffold Mule flows from a published API spec in Anypoint Code Builder (ACB)](https://www.prostdev.com/post/scaffold-mule-flows-from-published-api-spec-acb) Why would you want to scaffold Mule flows from an API specification? This way you will be able to get started on your Mule application with a base project that is created upon your specification, instead of starting the Mule project and Mule flows from scratch. Once you scaffold the Mule project, you will have: - Mule flows for each HTTP method in your specification - Basic error handling for different HTTP status codes - Initial Mule project with some Transform Message components where applicable ## Prerequisites - **Anypoint Platform** - You should have an Anypoint Platform account. You can create a new free trial account [here](https://anypoint.mulesoft.com/login/signup). - **API specification** - You should already have an API specification published in Anypoint Exchange. - **Anypoint Studio** - MuleSoft's IDE based on Eclipse. You can download it [here](https://www.mulesoft.com/lp/dl/anypoint-mule-studio). - **GitHub Repo** - If you want to follow along with the code I generated, you can check it out [here](https://github.com/ProstDev/codetober23/tree/main/day1). > [!DOCS] > If this is your first time creating an API specification, refer to [How to use MuleSoft's visual API Designer to create a To-Do API specification using clicks, not code](https://www.prostdev.com/post/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code) ## Create a Mule project in Studio First of all, let's open Anypoint Studio and click on **Create a Mule project** from the options on the left. ![Anypoint Studio Package Explorer with the "Create a Mule Project" link among project options](../../assets/blog/scaffold-mule-flows-from-published-api-specification-2.png) Add any name of your choice in **Project Name**. Click on the green ➕ button under **Import a published API** and select **from Exchange**. ![New project dialog: the add API button expanded to "from Exchange" or "from Maven"](../../assets/blog/scaffold-mule-flows-from-published-api-specification-3.png) Follow the prompts to authenticate to your Anypoint Platform account by clicking **Add Account** and adding your Username and Password. Once you are signed in, search for your API Specification, select it, and click on **Add >**. Once you selected it on the right-hand panel, click on **Finish**. ![Add Dependencies dialog with the To-Do API selected from Exchange before clicking Finish](../../assets/blog/scaffold-mule-flows-from-published-api-specification-4.png) After the API appears under **Import a published API**, you can click on **Finish** to close the window and see the generated project. ## Navigate the generated project Once the project has been created, you should be able to see the flows in the configuration file located under `src/main/mule`. ![Generated main flow with an HTTP Listener connected to an APIkit Router](../../assets/blog/scaffold-mule-flows-from-published-api-specification-5.png) The Error Handling has been added automatically to the main flow as well. This contains the HTTP status codes for some basic errors like 400 Bad Request or 404 Not Found. ![Auto-generated error handling with On Error Propagate scopes for BAD_REQUEST and NOT_FOUND](../../assets/blog/scaffold-mule-flows-from-published-api-specification-6.png) Notice how some of the flows already have a Transform Message component for you to see which DataWeave code is needed to retrieve some data like the URI Parameter(s). This, of course, is just in the case that your API Specification contains a URI Parameter. ![Generated get-by-id flow whose Transform Message reads attributes.uriParams.'id'](../../assets/blog/scaffold-mule-flows-from-published-api-specification-7.png) There will also be Transform Message components added for some cases where you have set up an example in the API Specification. ![Transform Message generated from an API example outputting sample JSON to-do data](../../assets/blog/scaffold-mule-flows-from-published-api-specification-8.png) Pretty cool, right? Keep posted for some more articles on scaffolding in Anypoint Studio or Anypoint Code Builder! Remember to subscribe at the bottom of the page to receive email notifications as soon as new content is published. Prost! --- ## #Codetober 2023 ~ 31 MuleSoft-related videos under 10min each! Source: https://prostdev.com/post/codetober-2023 | Published: Oct 3, 2023 | Category: News Here at [ProstDev](https://www.prostdev.com/), we dedicate ourselves to sharing knowledge with everyone who needs it. Because of that, every October we create this series of 31 videos ranging from various tech topics. > [!BUTTON] > [See YouTube Playlist](https://www.youtube.com/playlist?list=PLb61lESgk6hhKwJpM565UJVS5CxVZ4D1u) That's right...we'll publish ONE VIDEO A DAY! This year, the videos will be edited for your better digestion and will be less than 10 minutes each (hopefully). All of the scheduled videos will be listed below. If you can't access some of them yet, it is because they haven't been released as of now. Each video will be published at 12:00 a.m. ET. You can **subscribe** to our [YouTube channel](https://www.youtube.com/prostdev) to get notifications as soon as we publish new videos! Please comment on our videos to suggest future content or ask any questions about a specific topic (discussed in the video). Happy [#Codetober](https://www.prostdev.com/blog/hashtags/Codetober)! > [!TIP] > Find this year's GitHub repository here: [ProstDev/codetober23](https://github.com/ProstDev/codetober23) Remember to subscribe to our [YouTube channel](https://www.youtube.com/prostdev) to get notifications as soon as we publish new videos! Prost! --- ## How to use MuleSoft's visual API Designer to create a To-Do API specification using clicks, not code Source: https://prostdev.com/post/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code | Published: Sep 26, 2023 | Category: Tutorials Did you know MuleSoft provides this neat Graphical User Interface (GUI) where you can configure your own API specification with just clicks? That's right, you don't have to learn RAML or OAS anymore. You can just follow this visual tool to create them and it gives you the code you need in whichever format you need it! Let's create a To-Do API specification to see how this tool works. ## Prerequisites Before starting, create a new free trial account on [Anypoint Platform](https://anypoint.mulesoft.com/login/signup). Once you are logged in, Navigate to **Design Center**. You can access it using the left-hand menu or by clicking **Start designing** from the welcome screen. ## 1. Create a new API specification Inside Design Center, click on the **Create +** button and select **New API Specification**. Name your project however you prefer and make sure you select the **Guide me through it** option to use the visual tool. Click on **Create API**. ![New API specification dialog naming the project "My To-Do API" with the "Guide me through it" option selected.](../../assets/blog/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code-2.png) On the first screen, we'll see the API Summary. Add the following fields: ![API Summary with Title "My To-Do API", Version 1.0.0, and Media type set to application/json.](../../assets/blog/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code-3.png) ## 2. Create a data type Click on the plus (**+**) button next to the **DATA TYPES** tab on the left. Change the name to **To-Do** and the Type to **Object**. Next, click on the **Add Property** button and create the following properties for this object: > [!TIP] > Click on the Details (**v**) button to see the additional configurations for each property. ![To-Do object properties: id (Number), title (String), checked (Boolean, default False), dueDate (Date-Only).](../../assets/blog/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code-4.png) We are changing the default value for the *checked* property from *True* to *False* because normally when we create a new To-Do, it means that it hasn't been completed :-) ## 3. Create a /todos resource Click on the plus (**+**) button next to the **RESOURCES** tab on the left. Name the resource path **/todos**. ### 3.1. Read all To-Dos Under **GET > Responses** click **Add New Response** and leave the **200 - OK** option selected. Click on **Add Body** and change the type to **Array**. Open the details of the array and select Items Type **To-Do**. ![GET on /todos with a 200 OK response whose body is an Array of To-Do items.](../../assets/blog/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code-5.png) In other words, this means that whenever we send a GET request to the /todos resource (or path), we'll receive a 200-OK response with a list of our existing To-Dos. ### 3.2. Add Query Parameters Since we will be receiving a list of all the To-Dos, perhaps we could make things easier for the user by providing some filters for this array. We can do this by adding some Query Parameters to the GET method we just created. For this, go to **GET > Query Parameters** and click on **Add Query Parameter**. Add the following parameters: ![GET on /todos with four optional query parameters: titleContains, isChecked, fromDate, toDate.](../../assets/blog/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code-6.png) As you can see, these are all optional parameters. We do this by unchecking the Required option. Now the user can choose to either use these filters or not to retrieve a list of To-Dos. ### 3.3. Create a new To-Do To create a new To-Do in our design, we have decided to use the POST method under the /todos resource. For this, navigate to **POST > Responses** and click on **Add New Response**. You could leave this as 200 if you wanted to, but let's spice things up a bit by selecting **201 - Created** instead. You know, just for fun. Now click on **Add Body** and change the type to **To-Do**. ![POST on /todos with a 201 Created response whose body type is To-Do.](../../assets/blog/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code-7.png) This is not all! Now head to the **Body** tab right next to the Responses tab that we're in right now and click on **Add Body**. Change this type to **To-Do** as well. ![POST on /todos Body tab with one request body of type To-Do.](../../assets/blog/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code-8.png) This means that when the user wants to create a new To-Do, they'll have to send in a To-Do in the body of the request and they will receive a 201-Created with the details of the To-Do that was just created. > [!NOTE] > **❓ Question**: Can you see a potential problem for us to be using the To-Do data type in the creation of a To-Do? We'll discuss this at the end of the post :) ## 4. Create a /todos/{id} resource Click on the plus (**+**) button next to the **RESOURCES** tab on the left. Name the resource path **/todos/{id}**. > [!TIP] > You can also click on the plus button next to the /todos resource to automatically create a nested resource. ### 4.1. Add a URI Parameter Click on the **URI Parameters** button next to the resource path and select **Add URI Parameter**. Add the following values: ![Single to-do resource with a required URI parameter named id of type Number.](../../assets/blog/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code-9.png) This *id* will act as a sort of variable for the real value that will be passed in the URI. For example, to refer to the To-Do with id 1, you can call /todos/1. ### 4.2. Read a To-Do Under **GET > Responses** click **Add New Response** and leave the **200 - OK** option selected. Click on **Add Body** and change the type to **To-Do**. ![GET on a single to-do resource with a 200 OK response whose body type is To-Do.](../../assets/blog/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code-10.png) In this case, we are not creating any Query Parameters since we are solely returning one To-Do. The one that matches the id from the URI Parameter in the path. ### 4.3. Update a To-Do Under **PUT > Responses** click **Add New Response** and leave the **200 - OK** option selected. Click on **Add Body** and change the type to **To-Do**. ![PUT on a single to-do resource with a 200 OK response whose body type is To-Do.](../../assets/blog/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code-11.png) Now head to the **Body** tab right next to the Responses tab that we're in right now and click on **Add Body**. Change this type to **To-Do** as well. ![PUT on a single to-do resource, Body tab with one request body of type To-Do.](../../assets/blog/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code-12.png) This is awfully similar to the configurations we did for the POST method in the /todos resource, right? Honestly, we could've chosen to use POST here as well instead of PUT. It really depends on how you prefer to design your API specification. It's all very subjective sometimes! But I have an article for you on why I prefer to use PUT here instead of POST. Check it out 👇 > [!DOCS] > **Article**: [Understanding APIs (Part 3): What are HTTP Methods?](https://www.prostdev.com/post/understanding-apis-part-3-what-are-http-methods) ### 4.4. Delete a To-Do Under **DELETE > Responses** click **Add New Response** and select the **204 - No content** option. We won't add a body here in either the response or the request body. This is because we are already retrieving the id of the To-Do we want to delete (from the URI Parameter) and once we remove it, we won't have to return this information back in the response's body. ![DELETE on a single to-do resource with a 204 No Content response and no body.](../../assets/blog/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code-13.png) If you're curious about why I'm not just choosing 200-OK for all the status responses, you can check out this post I wrote about HTTP Status Codes 👇 > [!DOCS] > **Article**: [Understanding APIs (Part 6): What are HTTP Status Codes?](https://www.prostdev.com/post/understanding-apis-part-6-what-are-http-status-codes) ## Conclusion And that's all!! That's how you create an API specification for a To-Do app. I don't know if you've been noticing that there's some code being updated on the right-hand side of the screen. If you look at the bottom of that section, you'll notice there are two tabs: one that says RAML and one that says OAS. You can totally use this tool to generate a base of your RAML/OAS code so you don't have to start from scratch! Please note that there's an option to edit the code directly from the API Designer, however, if you do that, you'll lose the capability to come back to the GUI. If you intend to use this tool only to generate the code, I would recommend you download it and open it in a new API spec project instead of modifying it from here. Just in case you need this GUI in the future for reference :-) ## Answer to the question There was a question I dropped somewhere in the middle: - Can you see a potential problem for us to be using the To-Do data type in the creation of a To-Do? The answer is that because we created the To-Do data type to have a required property called *id,* and since we are using this data type to create a new To-Do, the id property will be required in the body of the request. For example, if we were to create a new To-Do, our request body would have to look like this: ```json { "id": 1, "title": "Example", "checked": false, "dueDate": "2023-09-25" } ``` When in reality, we wouldn't know what the appropriate id would be for this new To-Do. One should have to be assigned from the backend, the user shouldn't have to provide it. A solution. to this would be to go back to the POST method in the /todos resource and change the request body's data type from To-Do to Object. Then, we would have to manually add the rest of the properties here (title, checked, dueDate), which is not ideal. ![POST /todos body changed to an Object with manually re-added title, checked, and dueDate properties.](../../assets/blog/how-to-use-mulesoft-s-visual-api-designer-to-create-a-to-do-api-specification-using-clicks-not-code-14.png) If our initial To-Do data type changes, we would have to change the structure from both places. This will eventually lead to human error and there is no reusing in place for this. This is why the GUI is very helpful in generating an initial base of our API specification, but it currently doesn't hold all the answers to create the best RAML/OAS code. I still use it pretty much all the time to get started, though. Tools are exactly that: tools. They are meant to help you in your work. But they don't hold all the answers. We do :-) Prost! --- ## DataWeave scripts to clean your XML/HTML code snippets for a WordPress blog post Source: https://prostdev.com/post/dataweave-scripts-to-clean-your-xml-html-code-snippets-for-a-wordpress-blog-post | Published: Sep 5, 2023 | Category: Tutorials In case you're not familiar with my [dataweave-scripts](https://github.com/alexandramartinez/dataweave-scripts) GitHub repo, it's the place where I keep some of the scripts I've created to help the community with transformation questions or simply some scripts that have been handy to me. In this post, I want to introduce you to two transformations I added because of a use case I came up with last week. Basically to help clean an XML or HTML to publish a script in a WordPress article. ## The problem This problem started because I had written a blog post in a WordPress-based blog. I was sharing a Maven snippet (XML format). The issue is that WordPress mistook the XML tags as HTML code. So, instead of having a regular XML snippet, the article was showing something like this: ![Maven XML snippet in WordPress with its tags stripped out, leaving only the bare values](../../assets/blog/dataweave-scripts-to-clean-your-xml-html-code-snippets-for-a-wordpress-blog-post-2.png) The fix was simple. Instead of having the regular `<` and `>` characters pasted in the code snippet, I had to use `<` and `>` respectively. *(Thanks so much Julian Duque for providing the fix! I had no idea about this issue in WordPress* 🤗*)* For example, instead of writing ``, I had to replace it with `<plugin>` I thought to myself: If I need to keep doing this for future blog posts, maybe I can create a DataWeave transformation to fix this for me so I can just easily copy and paste the new clean snippet. These are the two approaches I came up with. ## First approach: XML input The first thing I tried to do since I was using an XML format for the script, was to take an input XML format, transform it to a String, and then clean the text. This is the script I came up with: ```dataweave %dw 2.0 output text/plain --- write(payload,"application/xml") replace "\n" with "" replace "<" with "<" replace ">" with ">" ``` > [!PLAYGROUND] > [Open in the Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=alexandramartinez%2Fdataweave-scripts&path=functions%2Fclean-xml) However, I quickly ran into issues when I tried to clean an HTML code snippet using this same transformation. This is how I came up with the second approach. ## Second approach: plain text input This time I decided to use a plain text input instead of an XML input format. This way, both XML and HTML code snippets could be used as the input and I wouldn't need to use the `write()` function in the first place. ```dataweave %dw 2.0 output text/plain --- payload replace "<" with "<" replace ">" with ">" ``` > [!PLAYGROUND] > [Open in the Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=alexandramartinez%2Fdataweave-scripts&path=functions%2Fclean-html) Plus, I got rid of one `replace()` because I no longer needed to remove the XML header. It's a short post, but I hope it's insightful for you all 🤗 I'm sure I'll keep using this example in the Playground to modify my WordPress posts in the future. Let me know if you've faced similar issues with WordPress before! --- ## Down the Rabbit Hole vol.2 - GUIs in MuleSoft Source: https://prostdev.com/post/down-the-rabbit-hole-vol-2-guis-in-mulesoft | Published: Aug 8, 2023 | Category: Opinion *GUIs, you love em or you hate em.* Software these days often gets shipped with a fancy user interface, though many developer tools are still CLI based. We devs seem to love the CLI. It is powerful and scares away the annoying managers and *The Business*, but most importantly: it works nicely with our scripts, builds, and pipelines! We do use GUIs though and cannot live without them, think of your APIKit Console, or Postman. Yes, once your collections are done you’ll end up running them with CLI variant Newman, but that’s for some other blog. No, What I want to cover today is GUIs in your integrations. 99/100 times this won’t be needed, but we’ll discuss those edge cases where a GUI might solve your problems. GUIs are there to help and visualize, but if you ask me there is a gap in the development lifecycle where we’re not used to providing them. We as developers are content to work with the console, we print DEBUG statements and ERRORs and can solve any problem with that. But that’s technical operations, what about the functional operations? Do I want to give the technical logs to the functional maintainer? Will they understand, or will they even check those logs regularly? As a developer I don’t mind checking if a batch has run, or which records failed, but if those questions arise regularly that quickly becomes a time sink. Now let's assume our business user is as non-technical as they get, they don’t want to see logs, even if we’ve limited those to the notifications intended for them. We can hook up an in-flow SMTP connector and send them a mail, or use a platform alert, but that might not be a good way of sharing the information they want. Let's discuss an alternative and some examples of that. I promise you, a well-designed interface can make your life as a Dev/Ops engineer more relaxed, and keep your focus on those tasks that actually matter. The first Mule GUI I ever made was years ago and used the static resource handler to display a simple HTML form, if you’re interested look at my horrible project here:[Mule Hackathon — RealLifeExploration Game](https://dev.to/wds444/real-life-exploration-game-powered-by-mulesoft-3bl6). It’s a mess of code and was made under a time crunch without all the dedicated software and services a real company has. I didn’t have a web server, so Mule had to do it. Same for the app logic, or the “Database”. Which was just a CSV file getting updated whenever an action happened. So far, I’ve only had to **work** with Mule GUIs four or five times. Only one of these was an actual customer integration, but two of them were deployed in production. I’m pretty sure you’ve even used one, maybe even in your CloudHub production VPC… Skeptical? I am of course talking about Professional service’s[net-tools API](https://github.com/mulesoft-consulting/net-tools-api)! ![net-tools Network Connectivity Tools web UI with a curl form and DNS console output](../../assets/blog/down-the-rabbit-hole-vol-2-guis-in-mulesoft-2.png) In the tried and trusted platform network testing toolkit, a client needs some connectivity tested? Use net-tools and curl to check if it’s working. Mule can’t resolve a Hostname? Use net-tools to check if the DNS server is available. > [!TIP] > Make sure to remove it once done, and don’t expose it to the internet for too long, if someone can find it they have some powerful tools available to check out your network. It offers some nice features, but what I'm more interested in is the code! ![DataWeave script serving static web files from classpath based on the requested URI path](../../assets/blog/down-the-rabbit-hole-vol-2-guis-in-mulesoft-3.png) The net-tools project has some iterations out there on the internet, but here we see the engine of this version’s ‘webserver’ any URL requested gets parsed and cross-referenced with a file in the web directory. That directory contains an index.html file and any necessary .js, .css, or other resources. The HTTP>Load Static Resource handler can do roughly the same but the result is that mule can serve a webpage on a chosen URL and port. These flows can use security filters, like net-tools is doing, or even be secured by API-Manager Policies and thus be available to any user you choose. In some of these projects, we took it a bit further, and we created specific Object stores for metadata, statistics, and even half-transformed payloads. Using Ajax in the HTML page can easily change this static display into a page that can dynamically display whatever info you choose to supply it from the endpoint. Now, we’ve discussed the HOW, now let’s look at the WHY. A practical example was a gigantic project where we had non-technical business users that wanted live insight into the tests they were doing. We’re talking about a few developers working on the APIs and dozens actively bombarding the system with tests. We could not handle the volume and logging all payloads was not feasible. The questions they had often had us look into the payload in different stages of processing. For reference, we’re talking input, JSON parsed input, two (hopefully identical) JSON responses, and a marshaled output. These messages were hundreds, sometimes thousands of lines long, and while we were not versed in reading them, they were. Had something like Splunk or ELK been available we could have used that, but it was not possible within the limitations. What we decided to do, at least during the development phase that lasted for over a year, was to store all these partially processed payloads into dedicated Object Stores with the correlation ID as their key. A five-minute job created a few endpoints that would return this data in text format. The interface from the net-tools API was *borrowed* and expanded with a few extra textboxes and UI elements. The dropdown box now used AJAX to populate the choices of correlation IDs and Ajax would GET these different payload values using the dedicated API endpoints. Construction of this interface from concept to deployment took less than a day, and enabled Functional Operations to do their work without interference from us! A few iterations of feedback later we would include some fancy CSS to mark those parts of the payload response that were not identical which was not too difficult either. Another feature was that we added a little metric stating the percentage of ‘correct’ requests which gave the entire functional team a great insight into the effectiveness and completion of the entire project. (parsing and processing 2000+ separate messages). The Business was extremely happy with this, as their work now became magnitudes easier. What started as a slight sidetrack to prevent a small annoyance is now used daily and really helps to show the total project progress! Another small example: The business was used to providing a monthly CSV file, but the FTP server they did that on had to go. A quick HTML page later that allowed them to upload that CSV file instead, to be loaded straight into the application logic solved all the requirements! (Yeah, I did not recommend this solution architecturally, but we were limited in the accepted options…) To sum up the post above: GUIs in Mule are really easy and can help you display data straight from the source. From testing while the project is in development, to easily displaying metadata or debug info to less than technical users. Even updating data or logic is possible, just make sure you properly think of the access controls and who you're providing them to. And don’t forget the most important rule: **Sanitize and verify any input!** ![My first ever Mule GUI…](../../assets/blog/down-the-rabbit-hole-vol-2-guis-in-mulesoft-4.png) --- ## DataWeave programming challenge #8: Sum all digits to get a 1-digit number Source: https://prostdev.com/post/dataweave-programming-challenge-8 | Published: Jul 11, 2023 | Category: Challenges Try to solve this challenge on your own to maximize learning. We recommend you refer to the [DataWeave documentation](https://docs.mulesoft.com/dataweave/latest/) **only**. Try to avoid using Google or asking others so you can learn on your own and become a DataWeave expert! > [!PLAYGROUND] > [Solve on the Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=alexandramartinez%2Fdataweave-challenges&path=challenges%2F8) ## Input Consider the following input payload (can be of **txt** format): ``` 2456 2 25235 2456 6 54 58795 3 456 76544 0 ``` ## Explanation of the problem Create a DataWeave script to sum all the digits from the input payload until there's only one digit left. For example, - 123 + 456 = 579 - 579 is broken down into 5 + 7 + 9 = 21 - 21 is broken down into 2 + 1 = 3 - The final result is 3 because it's a 1-digit number ## Expected output In this case, the expected output would be: ``` 2 ``` ## Clues If you're stuck with your solution, feel free to check out some of these clues to give you ideas on how to solve it! ### Clue #1 You can first sum each line from the input payload to end up with just one number. ### Clue #2 Use sizeOf() to get the length of a String. ### Clue #3 You can use reduce() to sum some of the digits. ## Answer If you haven't solved this challenge yet, we encourage you to keep trying! It's ok if it's taking longer than you thought. We all have to start somewhere ✨ Check out the clues and read the docs before giving up. You got this!! 💙 There are many ways to solve this challenge, but you can find some solutions we provide here so you can compare your result with us. ### Solution #1 ```dataweave %dw 2.0 import lines from dw::core::Strings output application/json fun sumNums(num, result=0) = do { var new = (num reduce $+$$) as String --- if (sizeOf(new) > 1) sumNums(new, result+new) else new } --- sumNums(sum(lines(payload))) as Number ``` ### Solution #2 ```dataweave %dw 2.0 import lines from dw::core::Strings output application/json fun sumNums(num, result=0) = do { var new = (num reduce $+$$) as String --- if (sizeOf(new) > 1) sumNums(new, result+new) else new } --- payload replace "\n" with "" then sumNums($) as Number ``` Subscribe to receive notifications as soon as new content is published ✨ --- ## We are now accepting guest writers! Source: https://prostdev.com/post/we-are-now-accepting-guest-writers | Published: Jun 28, 2023 | Category: News Hi ProstDev community! ✨ 🤗 We are delighted to announce an exciting update to our blog's article submission process! We've made significant improvements to enhance the experience for everyone involved and ensure the delivery of exceptional content to our valued readers. 📚 While our past contributions have predominantly focused on MuleSoft or DataWeave, it's important to note that our blog caters to a wide range of tech topics. Whether you have expertise in any technology, this platform welcomes your valuable insights and contributions! 🤓 We take immense pride in offering personalized 1:1 content reviews. Even if you're new to the world of blogging, rest assured that our dedicated team is here to support you every step of the way. Feel free to reach out with any inquiries or submit your article to initiate the review process. 🧐 Curious about our review process? It begins by acquainting yourself with our [guidelines](https://www.prostdev.com/post/guidelines-to-submit-your-content-in-prostdev) and [submitting](https://github.com/ProstDev/blog-posts/issues/new/choose) an article proposal. Once you've done that, our amazing team jumps in and makes some tweaks or suggestions to make your article more readable, understandable, and engaging for your audience. ⏳ Remember, practice makes perfect. No one starts out as an expert writer. We've all been there, taking those first steps. Our goal is to be your mentors and guide you along the way, helping you get comfortable with writing articles so you can rock it on your own! 📱 But wait, there's more! We don't just assist with your content. We've got a ready-to-learn fan base eager to dive into more technologies with your help! We'll take care of the article formatting, scheduling, and promotion stuff so that you can put all your focus on creating killer content. We can't wait to see what you'll [submit](https://github.com/ProstDev/blog-posts/issues/new/choose)! ✨ - The ProstDev team --- ## DataWeave programming challenge #7: Modify certain values from a JSON structure Source: https://prostdev.com/post/dataweave-programming-challenge-7 | Published: Jun 13, 2023 | Category: Challenges Try to solve this challenge on your own to maximize learning. We recommend you refer to the [DataWeave documentation](https://docs.mulesoft.com/dataweave/latest/) **only**. Try to avoid using Google or asking others so you can learn on your own and become a DataWeave expert! > [!PLAYGROUND] > [Solve on the Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=alexandramartinez%2Fdataweave-challenges&path=challenges%2F7) > [!TIP] > Instead of trying to write the whole code yourself, browse in the documentation for any function that would do the work for you 👀 ## Input Consider the following JSON input payload: ``` { "name": "a", "object": { "name": "b", "l": [ { "other": "c", "list": [ ] }, { "thisname": "def", "list": [ { "this": "500e", "l": [ { "finalname": "f" }, { "finalname": "ghijk" } ] } ] } ] }, "array": [ { "thisname": "h" }, { "xyz": "abc123" } ] } ``` ## Explanation of the problem Create a DataWeave script that will update all the values to uppercase, **except** the ones in which the field equals **thisname**. For example, the first field **name** with the value **"a"** has to be transformed to **"A"**. However, the field **thisname** with the value of **"def"** should stay the same. ## Expected output In this case, the expected output would be: ``` { "name": "A", "object": { "name": "B", "l": [ { "other": "C", "list": [ ] }, { "thisname": "def", "list": [ { "this": "500E", "l": [ { "finalname": "F" }, { "finalname": "GHIJK" } ] } ] } ] }, "array": [ { "thisname": "h" }, { "xyz": "ABC123" } ] } ``` ## Clues If you're stuck with your solution, feel free to check out some of these clues to give you ideas on how to solve it! ### Clue #1 Check out the [Tree](https://docs.mulesoft.com/dataweave/latest/dw-tree) module to see if any of those functions is useful for this case. ### Clue #2 Check out the [mapLeafValues](https://docs.mulesoft.com/dataweave/latest/dw-tree-functions-mapleafvalues) function. ### Clue #3 Use the [upper](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-upper) function to transform the strings to uppercase. ### Clue #4 Use **path[-1].selector** to retrieve the name of each value's field and compare it with a given string value. ## Answer If you haven't solved this challenge yet, we encourage you to keep trying! It's ok if it's taking longer than you thought. We all have to start somewhere ✨ Check out the clues and read the docs before giving up. You got this!! 💙 There are many ways to solve this challenge, but you can find here some solutions we are providing so you can compare your result with us. ### Solution #1 ```dataweave %dw 2.0 import mapLeafValues from dw::util::Tree output application/json --- payload mapLeafValues ((value, path) -> if (path[-1].selector == "thisname") value else upper(value) ) ``` ### Solution #2 ```dataweave %dw 2.0 import mapLeafValues from dw::util::Tree output application/json --- payload mapLeafValues if ($$[-1].selector == "thisname") $ else upper($) ``` Subscribe to receive notifications as soon as new content is published ✨ --- ## Part 5: CI/CD pipeline with MuleSoft and GitHub Actions - Enabling MFA through a Connected App Source: https://prostdev.com/post/part-5-ci-cd-pipeline-with-mulesoft-and-github-actions-enabling-mfa-through-a-connected-app | Published: May 16, 2023 | Category: Tutorials So far we’ve been setting up our CI/CD pipelines using our Anypoint Platform username and password. However, if you’re using an enterprise account, most likely you’re using MFA or Multi-Factor Authentication for your account. The process to create your CI/CD pipeline with this authentication method is quite different. If you haven’t been following the series or you’re not familiar with GitHub Actions, we recommend you start from the [first article](https://www.prostdev.com/post/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions) to understand how we are setting up all the configurations we need. In this post, we’ll learn how to set up our pipeline with a connected app, which is useful when you have multi-factor authentication activated in Anypoint Platform. ## Prerequisites You should already understand the basic CI/CD setup we’ve been doing in the previous articles. In summary, this is what you should already know: - How to configure and run the `build.yml` file for the pipeline under `.github/workflows`. - How to configure secrets in your GitHub repository. > [!NOTE] > The code for this article is **not** located on the **main** branch of the `github-actions` repository. Instead, a new branch was created to avoid confusion: [connected-app](https://github.com/alexandramartinez/github-actions/tree/connected-app). If you want to check the rest of the files we discuss here, like pom or build, please refer to that branch. ## Create a connected app in Anypoint Platform Because we are signing in to Anypoint Platform using MFA (through an app on your phone or an SMS code, for example), we can no longer rely on just our username and password. We have to create something called Connected App. To do this, sign in to [Anypoint Platform](https://anypoint.mulesoft.com/) and navigate to **Access Management** > **Connected Apps**. ![Access Management Connected Apps page with the Create app button and no apps yet](../../assets/blog/part-5-ci-cd-pipeline-with-mulesoft-and-github-actions-enabling-mfa-through-a-connected-app-2.png) Click on **Create app**. Add any name you want to identify this app, like `github-actions`. Select **App acts on its own behalf** and click on **Add Scopes**. Select the following scopes for your current business group and the **Sandbox** environment (since this is the one we’re using for this demo): - Design Center Developer - View Environment - View Organization - Profile - Cloudhub Organization Admin - Create Applications - Delete Applications - Download Applications - Read Applications - Read Servers ![Connected App scope selection listing Design Center, General, OpenID and Runtime Manager scopes](../../assets/blog/part-5-ci-cd-pipeline-with-mulesoft-and-github-actions-enabling-mfa-through-a-connected-app-3.png) Click on **Save**. After you create the app, make sure to copy both **ID** and **Secret**. We will use these in the pipeline for our authentication method. ![The created github-actions connected app with Copy Id and Copy Secret buttons](../../assets/blog/part-5-ci-cd-pipeline-with-mulesoft-and-github-actions-enabling-mfa-through-a-connected-app-4.png) In this case, for demonstration purposes, these are the credentials I’ll be using: ``` ID: bf51f105b644471f812b2e0c0cb8a97b Secret: 66B5ADCB18D4439D9C744236e3c590d3 ``` ## Set up your credentials on GitHub Just as we’ve done before, go to your GitHub repository and click on the **Settings** tab. Select **Secrets and variables** > **Actions** and add these two new secrets. - `CONNECTED_APP_CLIENT_ID` - `CONNECTED_APP_CLIENT_SECRET` The values should match what you previously extracted from Anypoint Platform. ## Modify your pom.xml In our `pom.xml` file, in the `org.mule.tools.maven` plugin, we used to have something like the following to authenticate via username and password. ```xml ... ${anypoint.username} ${anypoint.password} ``` We are going to replace the `username` and `password` fields with the following. ```xml ${client.id} ${client.secret} client_credentials ``` Here's the complete `pom.xml` for this example: ```xml 4.0.0 com.mycompany github-actions 1.0.0-SNAPSHOT mule-application github-actions UTF-8 UTF-8 4.4.0 3.8.0 amartinez17-github-actions Sandbox 2.3.13 org.apache.maven.plugins maven-clean-plugin 3.0.0 org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true https://anypoint.mulesoft.com ${app.runtime} ${client.id} ${client.secret} client_credentials ${app.name} ${env} MICRO us-east-2 1 true ${decryption.key} mule-application com.mulesoft.munit.tools munit-maven-plugin ${munit.version} test test test coverage-report ${decryption.key} true true 90 console sonar json html org.mule.connectors mule-http-connector 1.6.0 mule-plugin org.mule.connectors mule-sockets-connector 1.2.2 mule-plugin com.mulesoft.modules mule-secure-configuration-property-module 1.2.5 mule-plugin com.mulesoft.munit munit-runner 2.3.13 mule-plugin test com.mulesoft.munit munit-tools 2.3.13 mule-plugin test org.mule.weave assertions 1.0.2 test anypoint-exchange-v3 Anypoint Exchange https://maven.anypoint.mulesoft.com/api/v3/maven default mulesoft-releases MuleSoft Releases Repository https://repository.mulesoft.org/releases/ default mulesoft-releases MuleSoft Releases Repository default https://repository.mulesoft.org/releases/ false ``` That’s it for this file! Now let’s set up the pipeline to send these new credentials. ## Modify your build.yml Open the `build.yml` file we created inside `.github/workflows`. If you haven’t created this file yet, please refer to the [first article](https://www.prostdev.com/post/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions) to learn how to set it up. Go to the `deploy` job and locate the last step: **Deploy to Sandbox**. This is what we used to have under `env`: ```yaml USERNAME: ${{ secrets.anypoint_platform_username }} PASSWORD: ${{ secrets.anypoint_platform_password }} ``` And this is what we’ll replace it with: ```yaml ID: ${{ secrets.CONNECTED_APP_CLIENT_ID }} SECRET: ${{ secrets.CONNECTED_APP_CLIENT_SECRET }} ``` We’ll change the Maven command to match these new secrets/properties. Instead of sending username and password, like this: ```bash -Danypoint.username="$USERNAME" \ -Danypoint.password="$PASSWORD" \ ``` We’ll now send the app’s ID and secret. Like this: ```bash -Dclient.id="$ID" \ -Dclient.secret="$SECRET" \ ``` Here's the complete `build.yml` for this example: ```yaml name: Build and Deploy to Sandbox on: push: branches: [ main ] jobs: test: runs-on: ubuntu-latest steps: - name: Checkout this repo uses: actions/checkout@v3 - name: Cache dependencies uses: actions/cache@v3 with: path: ~/.m2/repository key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} restore-keys: | ${{ runner.os }}-maven- - name: Set up JDK 1.8 uses: actions/setup-java@v3 with: distribution: 'zulu' java-version: 8 - name: Test with Maven env: nexus_username: ${{ secrets.nexus_username }} nexus_password: ${{ secrets.nexus_password }} KEY: ${{ secrets.decryption_key }} run: mvn test --settings .maven/settings.xml -Dsecure.key="$KEY" - name: Upload MUnit reports uses: actions/upload-artifact@v3 with: name: munit-test-reports path: target/site/munit/coverage/ build: needs: test runs-on: ubuntu-latest steps: - name: Checkout this repo uses: actions/checkout@v3 - name: Cache dependencies uses: actions/cache@v3 with: path: ~/.m2/repository key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} restore-keys: | ${{ runner.os }}-maven- - name: Set up JDK 1.8 uses: actions/setup-java@v3 with: distribution: 'zulu' java-version: 8 - name: Build with Maven run: mvn -B package --file pom.xml -DskipMunitTests - name: Stamp artifact file name with commit hash run: | artifactName1=$(ls target/*.jar | head -1) commitHash=$(git rev-parse --short "$GITHUB_SHA") artifactName2=$(ls target/*.jar | head -1 | sed "s/.jar/-$commitHash.jar/g") mv $artifactName1 $artifactName2 - name: Upload artifact uses: actions/upload-artifact@v3 with: name: artifacts path: target/*.jar deploy: needs: build runs-on: ubuntu-latest steps: - name: Checkout this repo uses: actions/checkout@v3 - name: Cache dependencies uses: actions/cache@v3 with: path: ~/.m2/repository key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} restore-keys: | ${{ runner.os }}-maven- - uses: actions/download-artifact@v3 with: name: artifacts - name: Deploy to Sandbox env: ID: ${{ secrets.CONNECTED_APP_CLIENT_ID }} SECRET: ${{ secrets.CONNECTED_APP_CLIENT_SECRET }} KEY: ${{ secrets.decryption_key }} run: | artifactName=$(ls *.jar | head -1) mvn deploy -DskipMunitTests -DmuleDeploy \ -Dmule.artifact=$artifactName \ -Dclient.id="$ID" \ -Dclient.secret="$SECRET" \ -Ddecryption.key="$KEY" ``` ## Run the pipeline That’s it! Once you’re done with the changes, simply push a new change to the **main** branch and this will trigger the pipeline. ![GitHub Actions run with test, build and deploy jobs all succeeded after the push](../../assets/blog/part-5-ci-cd-pipeline-with-mulesoft-and-github-actions-enabling-mfa-through-a-connected-app-5.png) ## More resources You can check out my [GitHub profile](https://github.com/alexandramartinez) for more CI/CD repos: - [github-actions](https://github.com/alexandramartinez/github-actions) to deploy a Mule app to CloudHub - [dataweave-utilities-library](https://github.com/alexandramartinez/dataweave-utilities-library) to publish a DataWeave library to Exchange - [api-catalog-cli-example](https://github.com/alexandramartinez/api-catalog-cli-example) to update APIs in Exchange using the API Catalog CLI I hope this was helpful! Don't forget to subscribe so you don't miss any future content. --- ## DataWeave programming challenge #6: Using tail-recursion to get the factorial of a number Source: https://prostdev.com/post/dataweave-programming-challenge-6 | Published: May 9, 2023 | Category: Challenges Try to solve this challenge on your own to maximize learning. We recommend you refer to the [DataWeave documentation](https://docs.mulesoft.com/dataweave/latest/) **only**. Try to avoid using Google or asking others so you can learn on your own and become a DataWeave expert! > [!PLAYGROUND] > [Solve on the Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=alexandramartinez%2Fdataweave-challenges&path=challenges%2F6) ## Input Consider the following input payload (can be of **txt** format): ``` 3 5 -6 300 ``` ## Explanation of the problem - Create a [tail-recursive function](https://youtu.be/3UbTlLVNrVE) to get the [factorial](https://en.wikipedia.org/wiki/Factorial) of each **positive** number from the payload. - Sum the results. - Retrieve the digits/characters located at positions 20-25. - Make sure to return a number and not a string. - Numbers 0 or less won't be calculated. For example: - The factorial of 3 is 6 (3 x 2 x 1) - The factorial of 5 is 120 (5 x 4 x 3 x 2 x 1) - The factorial of -6 won't be calculated so the result will be 0 - And so on - Once the results are summed (6 + 120 + 0...), extract the digits at indexes 20 to 25. - Return this last 6-digit number. ## Expected output In this case, the expected output would be: ``` 537046 ``` ## Clues If you're stuck with your solution, feel free to check out some of these clues to give you ideas on how to solve it! ### Clue #1 Use lines() from the Strings module or splitBy() to process every line. ### Clue #2 Make sure to use tail-recursiveness and not regular recursiveness. Otherwise, your script might not work. ### Clue #3 Make sure to create an if statement before calling the function in recursion. Otherwise, you might get stuck in an infinite loop! ### Clue #4 You can use the keyword `as` to transform to/from strings and numbers. ### Clue #5 You can extract characters from a string with the range selector [ x to y ] ## Answer If you haven't solved this challenge yet, we encourage you to keep trying! It's ok if it's taking longer than you thought. We all have to start somewhere ✨ Check out the clues and read the docs before giving up. You got this!! 💙 There are many ways to solve this challenge, but you can find here one of my solutions! ### Solution #1 ```dataweave %dw 2.0 import lines from dw::core::Strings output application/json fun factorial(num:Number, result:Number=1):Number = do { if (num == 0) result else if (num < 0) 0 else factorial(num - 1, result * num) } --- lines(payload) map factorial($) then sum($) as String then $[20 to 25] as Number ``` Subscribe to receive notifications as soon as new content is published ✨ --- ## Part 4: CI/CD pipeline with MuleSoft and GitHub Actions - MUnit minimum coverage percentage Source: https://prostdev.com/post/part-4-ci-cd-pipeline-with-mulesoft-and-github-actions-munit-minimum-coverage-percentage | Published: May 2, 2023 | Category: Tutorials In the [previous article](https://www.prostdev.com/post/part-3-ci-cd-pipeline-with-mulesoft-and-github-actions-munit-testing), we learned how to set up our Nexus credentials to be able to run the MUnits from the CI/CD pipeline. If you haven’t been following the series or you’re not familiar with GitHub Actions, we recommend you start from the [first article](https://www.prostdev.com/post/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions) to understand how we are setting up all the configurations we need. In this post, we’ll see the steps to add the minimum coverage percentage for the MUnit tests in order to pass the pipeline. ## Prerequisites You should already have all the setup in your Mule application for running the MUnits in a CI/CD pipeline. In summary, this is what you should already have: - The required dependencies, plugins, or properties in your `pom.xml` to be able to run MUnits for your Mule project. - The working MUnit tests under `src/test/munit`. - A working CI/CD pipeline. See the [previous article](https://www.prostdev.com/post/part-3-ci-cd-pipeline-with-mulesoft-and-github-actions-munit-testing) if you still need to set this up. ## Modify your pom.xml This time there’s nothing to add/modify in the other files (`build.yml` / `settings.xml`). We’ll simply add the following configuration to our `pom.xml` under the existing `munit-maven-plugin` configuration. ```xml com.mulesoft.munit.tools munit-maven-plugin ${munit.version} test test test coverage-report ${decryption.key} true true 90 console sonar json html ``` Let’s explore each of the changes one by one. Starting with the `environmentVariables`. ```xml ${decryption.key} ``` We are adding this variable right here because it needs to be passed down to the Mule application to be able to decrypt the existing secured properties. > [!NOTE] > If you’re not sure how to do this setup or want to understand where we are using the `secure.key` or `decryption.key`, please refer to [Part 2](https://www.prostdev.com/post/part-2-ci-cd-pipeline-with-mulesoft-and-github-actions-secured-encrypted-properties) of this series. Next, we added the configuration to set the minimum coverage percentage for the application to pass the build. ```xml true 90 ``` In this case, we only set up the `requiredApplicationCoverage` option with a minimum percentage of 90. If you instead (or additional to) want to set up the percentage for the resources or the flows, you can use the other two options. To see the difference in these settings, you can refer to [this live stream](https://youtu.be/Z0ORL0yoKCo) or the [official documentation](https://docs.mulesoft.com/munit/latest/coverage-maven-concept#set-up-a-minimum-coverage-level). Finally, you can generate different coverage reports to be able to see your statistics in several formats. ```xml console sonar json html ``` You can choose to generate some of them or all of them. Simply add/remove the `format` field from this configuration. > [!DOCS] > For more information, see [Coverage Report Types](https://docs.mulesoft.com/munit/latest/coverage-maven-concept#coverage-report-types). > [!NOTE] > If you want to be able to see your generated reports from the CI/CD pipeline, you can add the following configuration under the `test` job in your `build.yml` file. Add this as the last step. You will be able to download the reports after the whole pipeline has finished running (see the next screenshot). ```yaml - name: Upload MUnit reports uses: actions/upload-artifact@v3 with: name: munit-test-reports path: target/site/munit/coverage/ ``` ## Run the pipeline That’s it! Once you’re done with the changes, simply push a new change to the **main** branch and this will trigger the pipeline. ![Successful GitHub Actions run with test, build, and deploy jobs and two artifacts](../../assets/blog/part-4-ci-cd-pipeline-with-mulesoft-and-github-actions-munit-minimum-coverage-percentage-2.png) ## More resources You can check out my [GitHub profile](https://github.com/alexandramartinez) for more CI/CD repos: - [github-actions](https://github.com/alexandramartinez/github-actions) to deploy a Mule app to CloudHub - [dataweave-utilities-library](https://github.com/alexandramartinez/dataweave-utilities-library) to publish a DataWeave library to Exchange - [api-catalog-cli-example](https://github.com/alexandramartinez/api-catalog-cli-example) to update APIs in Exchange using the API Catalog CLI I hope this was helpful! Don't forget to subscribe so you don't miss any future content. --- ## Down the Rabbit Hole vol.1 - Using GPUs in MuleSoft Source: https://prostdev.com/post/down-the-rabbit-hole-vol-1-using-gpus-in-mulesoft | Published: Apr 25, 2023 | Category: Opinion Hello, fellow tech enthusiasts! My name is Nicky, and I’m a MuleSoft expert developer and architect with six years of experience in the field. I’m also a recognized MuleSoft mentor and a big fan of learning by doing weird stuff. Recently, I’ve been working on a pet project that I thought might interest some of you. I’ve written a Mule plugin that has the ability to use **Cuda cores for parallel asynchronous processing**. Now, I know what you’re thinking: “That’s not recommended or supported!” And you’re right. This is not something that should be done in a production environment. However, in theory, and practice, this idea works. So why did I create this plugin? Well, I’m always on the lookout for new and innovative ways to push the boundaries of what’s possible with MuleSoft. And using Cuda cores for parallel processing is certainly one way to do that. ![Logo for the Mcuda Mule GPU Acceleration plugin](../../assets/blog/down-the-rabbit-hole-vol-1-using-gpus-in-mulesoft-2.png) Think about it: with the ability to leverage Cuda cores, we can achieve incredible levels of processing speed and power. For example, we’ve been trying to train a neural network for payload validation, which so far has had some positive results. Image processing, video encoding, and other intensive tasks could also benefit from this kind of parallel processing. Of course, there are some risks involved. If not done properly, using Cuda cores could cause instability, crashes, and other issues. That’s why I want to emphasize that **this is not something that you should start using in your projects**. However, for those who are interested in exploring the possibilities of this kind of technology, it’s definitely worth a read. ## Prerequisites We won’t dive too deep into the technical details of how this plugin works, it’s important to note that it **requires a machine with an NVidia GPU**. Without this hardware, it won’t be possible to utilize Cuda cores for parallel processing. Who knows; maybe we’ll see GPU-enabled Mule Runtimes in the future? Assuming you have the necessary hardware, the next step is to **install the developer packages for Cuda**. These packages include the necessary tools and libraries to program the Java code that will interact with the GPU. Once you have the Cuda developer packages installed, you can then utilize JCuda to communicate directly with the GPU memory. JCuda is a Java binding for the Cuda runtime and driver API, which makes it possible to write Java code that can take advantage of Cuda cores. (Runtime API = Basic, pre-compiled functionality. Driver API = write your own kernel, almost like low-level C programs.) ## Mule project To get started with JCuda, you’ll need to include the JCuda JAR files in your Mule project. You can download these JAR files from the JCuda website. Once you have the JCuda JAR files included in your project, you can start writing Java code that will utilize the GPU. > [!BUTTON] > [JCuda Tutorial](http://www.jcuda.org/tutorial/TutorialIndex.html) ![Java code using JCuda to allocate managed GPU memory and run a kernel from a Mule flow](../../assets/blog/down-the-rabbit-hole-vol-1-using-gpus-in-mulesoft-3.png) In conclusion, I’m excited to see where this kind of innovation could take us in the world of MuleSoft. As always, I encourage everyone to keep pushing the limits and exploring new ideas. Who knows what we might discover next? The below image shows a proof of concept interaction of a Mule flow and the GPU. Ignore the memcpy error, that’s my fault ;-) ![Anypoint Studio hello-cuda flow and console printing Hello test from CUDA from the GPU](../../assets/blog/down-the-rabbit-hole-vol-1-using-gpus-in-mulesoft-4.png) --- ## 5 tips to have a better setup when you're live streaming from a MacOS computer Source: https://prostdev.com/post/5-tips-to-have-a-better-setup-when-you-re-live-streaming-from-a-macos-computer | Published: Apr 19, 2023 | Category: Tutorials I've been doing a lot of live streaming lately. Either alone or with guests. And I noticed some things that could be improved (from a streamer perspective) to create a better experience for the people watching our screen. I'll give you some MacOS-specific tools, but I'm sure you can find a replacement for these for other operating systems. Let's take a look! ## 1. Adjust your screen's resolution This applies to any kind of screen, regardless of the operating system. Before starting the stream, whether you're using your own laptop/computer or an external monitor for screen-sharing, make sure you adjust the screen to a **1080 or 720 pixels resolution** (if you want Full HD or HD resolution). The reason behind this is that everything on the screen will look bigger and clearer in the live stream. A lot of people watch live streams on smaller screens. Don't assume that everyone will be able to just keep your stream on a full screen. Try to show your tools and other things as big as possible without pixelating the image and also keeping a high resolution. Here are some examples of why this setting is important. Look at the next two screens. Which one looks better for you? Can you read everything on the screen? ![Higher resolution, more space for the streamer, smaller/more difficult to see for the viewer.](../../assets/blog/5-tips-to-have-a-better-setup-when-you-re-live-streaming-from-a-macos-computer-2.png) ![Lower resolution, less space for the streamer, bigger/easier to see for the viewer.](../../assets/blog/5-tips-to-have-a-better-setup-when-you-re-live-streaming-from-a-macos-computer-3.png) Both are exactly the same screen, just with different resolution settings. The same effect will be on your live stream. It may be more comfortable for you to stream in your regular setting - which might be something similar to the first screenshot - but it will be harder for the viewers to follow/understand exactly what you're doing. You don't have to use the lowest resolution possible if it's uncomfortable for you or makes it harder to present the live coding on the screen. But try to adjust this setting where it feels better for you and also keeps a bigger image. Like the following picture. ![macOS desktop with a code editor and calculator app beside the System Settings Displays panel](../../assets/blog/5-tips-to-have-a-better-setup-when-you-re-live-streaming-from-a-macos-computer-4.png) To do this from a MacOS computer, go to your **System Settings** > **Displays**. Now, depending on whether you're using the actual MacOS computer or an external monitor for screen sharing, you will have different options. ### Option 1 - MacOS computer/laptop Change your screen resolution by selecting a larger text resolution from the thumbnails. You may have to play around with these a bit to make sure the image is good and you can still do your demo comfortably. ![macOS Displays resolution thumbnails from Larger Text to More Space, with Default selected](../../assets/blog/5-tips-to-have-a-better-setup-when-you-re-live-streaming-from-a-macos-computer-5.png) ### Option 2 - External monitor Change your screen resolution by selecting either **1080p** or **720p**. - **1080** will give you a higher resolution (Full HD or FHD) but will result in larger data (more GB for the video size). If your internet connection is not very good when live streaming, you can try changing to 720. If the data size and internet are not a problem, you can use 1080. - **720** will still be high-definition (HD) but won't be as clear as FHD. Normally these differences are not as easily caught if watched from a phone or a computer. They present more when watching from a larger device like a TV. ![macOS external-monitor resolution list with 1080p highlighted among options like 720p](../../assets/blog/5-tips-to-have-a-better-setup-when-you-re-live-streaming-from-a-macos-computer-6.png) If you want to see these differences yourself, you can try watching [this video](https://youtu.be/bsM4D-ek-T4) (or any video) in different resolutions. To do this, inside the video, go to the **settings** button > **quality**, and select either **1080** or **720**. > [!NOTE] > [That video](https://youtu.be/bsM4D-ek-T4) was recorded on an external monitor using the 1080p setting. ## 2. Zoom in on available applications There are some applications that can natively zoom in on the UI. For example, Visual Studio Code or Sublime Text. You can press **cmd +** to zoom in and **cmd -** to zoom out. In some applications, you might have to press **cmd =** to zoom in, depending on your settings and your keyword. For example, the following screenshot is how I normally use Visual Studio Code when I'm not sharing my screen. For you as the viewer, it might not be as clear to be able to follow what I'm doing. Even though I can see it as perfectly normal from my perspective. ![My normal view, without zooming in.](../../assets/blog/5-tips-to-have-a-better-setup-when-you-re-live-streaming-from-a-macos-computer-7.png) But if I press **cmd +**, I'll be able to zoom the UI in. It might look unnecessarily big from my perspective, but it's the best view for you who are just watching my shared screen. ![The UI after zooming in.](../../assets/blog/5-tips-to-have-a-better-setup-when-you-re-live-streaming-from-a-macos-computer-8.png) This looks way better for you now, right? And same as before, you don't have to make the screen incredibly big. Just find a balance where it's comfortable for you to work on it, but it's as big as possible for the people who are watching. ## 3. Use the accessibility settings Inside your **System Settings** > **Accessibility** > **Zoom**, there are several options to be able to use the Zoom tool on your screen. This is especially useful when you have some applications that don't support the 'standard' zooming options (**cmd +** or **cmd -**). ![macOS Accessibility Zoom settings with scroll-gesture zoom and Picture-in-Picture style enabled](../../assets/blog/5-tips-to-have-a-better-setup-when-you-re-live-streaming-from-a-macos-computer-9.png) The option that I always use for my streams is the **Picture-in-Picture** zoom. I have it set up to use the **command** key to be able to use the scroll from my mouse to zoom in/out. You can also click on the **Size and Location** button to set up how big/small you want the zoom to be, or click on the **Advanced** button for even more customization on the functionality. These are my Advanced settings, in case you want to set it up as I have it: ![Advanced options 1/2](../../assets/blog/5-tips-to-have-a-better-setup-when-you-re-live-streaming-from-a-macos-computer-10.png) ![Advanced options 2/2](../../assets/blog/5-tips-to-have-a-better-setup-when-you-re-live-streaming-from-a-macos-computer-11.png) ## 4. Display your clicks Your audience might not know if you're double-clicking or drag-and-dropping with your mouse. They might be able to hear through the microphone when you're clicking your mouse, but it's not good enough to let them guess what you're doing. You can use some software to show your clicks visually on-screen. The one I use is [keycastr](https://github.com/keycastr/keycastr). It shows a black line around the cursor to show if I just clicked once, twice, or if I'm pressing down on it. After you install it, make sure you select the **Display mouse clicks** checkbox to show them. Otherwise, they won't be showing. ![keycastr Display settings](../../assets/blog/5-tips-to-have-a-better-setup-when-you-re-live-streaming-from-a-macos-computer-12.png) I also use this application to display my keystrokes, which is my next and final point. ## 5. Display your keystrokes Sometimes you use keystroke combinations that the audience might not be aware of. For example, in Visual Studio Code, you have to press **cmd + shift + P** to show the palette. Your audience may not be aware of this combination and they just see you're magically opening the palette on the screen. Instead of letting your audience guess what you're doing, you can visually display what keystrokes you're pressing. There are many paid tools that are fancier and have more customization options, but I feel [keycastr](https://github.com/keycastr/keycastr) gets the job done - and it's free. After you install it, make sure you select the **Command keys only** checkbox. Otherwise, it will show ALL of your keystrokes. Including every single letter you press when writing - which can be annoying. But with this setting, it'll only show the combinations like **cmd + shift + P** or **cmd +**. ![keycastr Display settings](../../assets/blog/5-tips-to-have-a-better-setup-when-you-re-live-streaming-from-a-macos-computer-13.png) That's all for this post! I hope this is useful :D --- ## DataWeave programming challenge #5: Reverse a phrase's words, but keep the punctuation Source: https://prostdev.com/post/dataweave-programming-challenge-5 | Published: Apr 18, 2023 | Category: Challenges > [!NOTE] > This challenge is based on [Codeacademy's Reverse Words challenge](https://discuss.codecademy.com/t/challenge-reverse-words/83796) Try to solve this challenge on your own to maximize learning. We recommend you refer to the [DataWeave documentation](https://docs.mulesoft.com/dataweave/latest/) **only**. Try to avoid using Google or asking others so you can learn on your own and become a DataWeave expert! > [!PLAYGROUND] > [Solve on the Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=alexandramartinez%2Fdataweave-challenges&path=challenges%2F5) ## Input Consider the following input payload (can be of **txt** format): ``` Hello world May the Fourth be with you Hello world! With you, be May the Fourth With you, be May the Fourth! ``` ## Explanation of the problem Create a DataWeave script that will reverse the order of each one of the words in a phrase to form a new phrase. Each phrase or sentence is separated by a new line. Each word is separated by a space. However, the punctuation signs (! and ,) will have to stay in the same place. Lower/upper cases can be kept as-is. For example: - "Hello world!" becomes "world Hello!" - "With you, be May the Fourth!" becomes "Fourth the, May be you With!" ## Expected output In this case, the expected output would be: ``` [ "world Hello", "you with be Fourth the May", "world Hello!", "Fourth the, May be you With", "Fourth the, May be you With!" ] ``` > [!NOTE] > You can keep the output as a JSON or as a text, whatever your preference is. ## Clues If you're stuck with your solution, feel free to check out some of these clues to give you ideas on how to solve it! ### Clue #1 Keep the index of each punctuation sign so you can insert it there later ### Clue #2 You can use the isAlphanumeric function from the Strings module ### Clue #3 You can use the scan function to separate each phrase in an Array of Strings ### Clue #4 You can use the joinBy function to transform from Array to String ## Answer If you haven't solved this challenge yet, we encourage you to keep trying! It's ok if it's taking longer than you thought. We all have to start somewhere ✨ Check out the clues and read the docs before giving up. You got this!! 💙 There are many ways to solve this challenge, but you can find here my solution. I'm sure you all can make it better! :) Mine is super long 😂 ### Solution #1 ```dataweave %dw 2.0 import slice from dw::core::Arrays import lines, isAlphanumeric from dw::core::Strings output application/json --- lines(payload) map ((line) -> flatten( line scan /[^\w\d\s]|[\w]*/ ) filter (!isEmpty($))) map ((words) -> do { var wordsobj = (words map { word: $, originalIndex: $$, move: isAlphanumeric($) }) var reversed = (wordsobj filter ($.move))[-1 to 0] var spechars = wordsobj filter (not $.move) var endingStr = if (spechars[-1].originalIndex != null) (reversed[spechars[-1].originalIndex to -1].word default [] joinBy " ") else "" --- if (isEmpty(spechars)) reversed.word joinBy " " else spechars map ((charobj,coidx) -> slice( reversed, if (coidx-1 >= 0) spechars[coidx-1].originalIndex else 0, charobj.originalIndex ).word joinBy " " then trim("$($)$(charobj.word) $(endingStr)") ) joinBy " " }) ``` I also recorded myself coming up with the solution, but without explanations. Just some lo-fi music and a screen :) you can leave the video in the background while working! :D let me know if you find this useful. Subscribe to receive notifications as soon as new content is published ✨ --- ## DataWeave programming challenge #4: Solve the Tower of Hanoi mathematical puzzle Source: https://prostdev.com/post/dataweave-programming-challenge-4 | Published: Mar 21, 2023 | Category: Challenges > [!NOTE] > This challenge is based on the [Tower of Hanoi](https://en.wikipedia.org/wiki/Tower_of_Hanoi) mathematical puzzle - **this is a tough one!** Try to solve this challenge on your own to maximize learning. We recommend you refer to the [DataWeave documentation](https://docs.mulesoft.com/dataweave/latest/) **only**. Try to avoid using Google or asking others so you can learn on your own and become a DataWeave expert! > [!PLAYGROUND] > [Solve on the Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=alexandramartinez%2Fdataweave-challenges&path=challenges%2F4) > [!IMPORTANT] > There are 3 different scenarios to test your code is working properly and to ensure there are no hardcoded values. However, only the first scenario is provided in the Playground link below. Please copy+paste the other two scenarios to make sure your solution is working properly. ## Input Consider the following JSON inputs: **SCENARIO 1** ``` { "moves": 0, "disks": 3, "targetTower": "C", "towers": { "A": [ 1, 2, 3 ], "B": [], "C": [] } } ``` **SCENARIO 2** ``` { "moves": 0, "disks": 4, "targetTower": "C", "towers": { "A": [ 1, 2, 3, 4 ], "B": [], "C": [] } } ``` **SCENARIO 3** ``` { "moves": 0, "disks": 7, "targetTower": "Y", "towers": { "X": [], "Y": [], "Z": [ 1, 2, 3, 4, 5, 6, 7 ] } } ``` ## Explanation of the problem Create a DataWeave script to solve the [Tower of Hanoi](https://en.wikipedia.org/wiki/Tower_of_Hanoi) puzzle. The input payload contains the number of moves (starting at 0), the total number of disks in the game, the target tower where all the disks have to be moved, and the 3 towers containing each disk. (You can use [this link](https://www.mathsisfun.com/games/towerofhanoi.html) to play the game online) Here are the rules of the game: - With each move, you can only move one disk at a time from its current stack to a different stack. - You can only place smaller disks on top of bigger disks. For example, you can place disk 1 on top of disk 3 but you can't place disk 5 on top of disk 2. - You can only move the disk at the top of any stack. ## Expected output For each of the three scenarios from the input, these are the expected outputs: **SCENARIO 1** ``` { "moves": 7, "disks": 3, "targetTower": "C", "towers": { "A": [ ], "B": [ ], "C": [ 1, 2, 3 ] } } ``` **SCENARIO 2** ``` { "moves": 15, "disks": 4, "targetTower": "C", "towers": { "A": [ ], "B": [ ], "C": [ 1, 2, 3, 4 ] } } ``` **SCENARIO 3** ``` { "moves": 127, "disks": 7, "targetTower": "Y", "towers": { "X": [ ], "Y": [ 1, 2, 3, 4, 5, 6, 7 ], "Z": [ ] } } ``` ## Clues If you're stuck with your solution, feel free to check out some of these clues to give you ideas on how to solve it! ### Clue #1 Read the [wikipedia page](https://en.wikipedia.org/wiki/Tower_of_Hanoi) to have more context from the algorithm perspective. ### Clue #2 From the wikipedia page, take special attention to this part: For an even number of disks: make the legal move between pegs A and B (in either direction), make the legal move between pegs A and C (in either direction), make the legal move between pegs B and C (in either direction), repeat until complete. For an odd number of disks: make the legal move between pegs A and C (in either direction), make the legal move between pegs A and B (in either direction), make the legal move between pegs B and C (in either direction), repeat until complete. ### Clue #3 If the previous clue wasn't what you needed, take a look at these other rules from the wikipedia page: Number the disks 1 through n (largest to smallest). If n is odd, the first move is from peg A to peg C. If n is even, the first move is from peg A to peg B. Now, add these constraints: No odd disk may be placed directly on an odd disk. No even disk may be placed directly on an even disk. There will sometimes be two possible pegs: one will have disks, and the other will be empty. Place the disk on the non-empty peg. Never move a disk twice in succession. ### Clue #4 If you see other solutions (in Python, C++, Java, etc) remember that those don't necessarily apply to DataWeave. Try to come up with your own solution instead. ## Answer If you haven't solved this challenge yet, we encourage you to keep trying! It's ok if it's taking longer than you thought. We all have to start somewhere ✨ Check out the clues and read the docs before giving up. You got this!! 💙 There are many ways to solve this challenge, but you can find here my solution. I'm sure you all can make it better! :) Mine is super long 😂 ### Solution ```dataweave %dw 2.0 import update from dw::util::Values import drop, take from dw::core::Arrays output application/json var towerNames:Array = namesOf(payload.towers) var targetTower:String = payload.targetTower var initialTower:String = payload.towers pluck (if (!isEmpty($)) $$ else "") filter ($ != "") then $[0] as String var auxTower:String = (towerNames -- [targetTower, initialTower])[0] fun moveDisk(towers:Object, fromTower:String, toTower:String) = do { var from = towers[fromTower] var to = towers[toTower] --- towers update log("Moved from", fromTower) with (from drop 1) update log("to",toTower) with (from take 1)[0] >> to } fun swapDisks(towers:Object, tower1:String, tower2:String) = do { var disk1 = towers[tower1][0] default (payload.disks + 1) var disk2 = towers[tower2][0] default (payload.disks + 1) --- if (disk1 < disk2) moveDisk(towers, tower1, tower2) else moveDisk(towers, tower2, tower1) } fun play(game:Object, moveNumber:Number = 1) = do { var isEvenGame = isEven(game.disks) var isOddGame = isOdd(game.disks) @Lazy var newTowers = if ( (isEvenGame and moveNumber == 1) or (isOddGame and moveNumber == 2) ) swapDisks(game.towers, initialTower, auxTower) else if ( (isEvenGame and moveNumber == 2) or (isOddGame and moveNumber == 1) ) swapDisks(game.towers, initialTower, targetTower) else swapDisks(game.towers, auxTower, targetTower) @Lazy var newMoveNumber = if (moveNumber == 3) 1 else moveNumber + 1 var newGame = game update "towers" with newTowers update "moves" with game.moves + 1 --- if (sizeOf(game.towers[targetTower]) == game.disks) game else play(newGame, newMoveNumber) } --- play(payload) ``` I also recorded myself coming up with the solution, but without explanations. Just some lo-fi music and a screen :) you can leave the video in the background while working! :D let me know if you find this useful. Subscribe to receive notifications as soon as new content is published ✨ --- ## Part 3: CI/CD pipeline with MuleSoft and GitHub Actions - MUnit testing Source: https://prostdev.com/post/part-3-ci-cd-pipeline-with-mulesoft-and-github-actions-munit-testing | Published: Mar 14, 2023 | Category: Tutorials In the [previous article](https://www.prostdev.com/post/part-2-ci-cd-pipeline-with-mulesoft-and-github-actions-secured-encrypted-properties), we learned how to add a decryption key to your CI/CD pipeline to be able to decrypt your encrypted properties at runtime. If you haven’t been following the series or you’re not familiar with GitHub Actions, we recommend you start from the [first article](https://www.prostdev.com/post/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions) to understand how we are setting up all the configurations we need. In this post, we’ll see the steps to create a pipeline with GitHub Actions that will run the MUnit tests you have set up in your Mule project. > [!NOTE] > We will talk about coverage percentage in the next article. We’ll only focus on creating the basic testing configuration in this post. ## Prerequisites You should already have all the setup in your Mule application for running the MUnits. In summary, this is what you should already have: - The required dependencies, plugins, or properties in your `pom.xml` to be able to run MUnits for your Mule project. - The working MUnit tests under `src/test/munit`. - Enterprise Nexus credentials to run the MUnits in the CI/CD pipeline. For more information see [this question](https://help.mulesoft.com/s/question/0D52T000064SVU4SAO/anypoint-platform-credentials-and-nexus). Note that we didn’t need these credentials before but we need them to be able to run the tests because of some dependencies. > [!IMPORTANT] > Make sure your MUnit tests work locally (from Anypoint Studio) before attempting to create the pipeline. Otherwise, you might encounter some issues. > [!NOTE] > You do not need Nexus credentials to run the MUnits from Studio, only to run them with Maven. ## Set up your credentials In your GitHub repository, go to the **Settings** tab (make sure you are signed in to see it). Now go to **Secrets and variables > Actions**. Here you will be able to set up your repository secrets. In the previous articles, we added the following actions secrets: - `ANYPOINT_PLATFORM_PASSWORD` - `ANYPOINT_PLATFORM_USERNAME` - `DECRYPTION_KEY` For this post, we’re going to add two more secrets for the nexus credentials: - `NEXUS_USERNAME` - `NEXUS_PASSWORD` Click on **New repository secret**. In the Name field, write the name of the secret. In the Secret field, write the actual value of your key (your Nexus username/password). Click **Add secret**. Please make sure your Nexus credentials are valid before continuing. To verify this, go to [this site](https://repository-master.mulesoft.org/nexus/#welcome) and sign in. ![Nexus Repository Manager sign-in dialog prompting for username and password](../../assets/blog/part-3-ci-cd-pipeline-with-mulesoft-and-github-actions-munit-testing-2.png) You should be able to see the welcome page after logging in. ![Nexus Repository Manager welcome page shown after a successful login](../../assets/blog/part-3-ci-cd-pipeline-with-mulesoft-and-github-actions-munit-testing-3.png) ## Set up your build.yml In the last posts, we set up and modified our `build.yml` file under `.github/workflows`. This time, we are going to be using the same base file, just adding a few modifications to include the new configuration. You can copy and paste the following code into this file. ```yaml name: Build and Deploy to Sandbox on: push: branches: [ main ] pull_request: branches: [ main ] jobs: test: runs-on: ubuntu-latest steps: - name: Checkout this repo uses: actions/checkout@v3 - name: Cache dependencies uses: actions/cache@v3 with: path: ~/.m2/repository key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} restore-keys: | ${{ runner.os }}-maven- - name: Set up JDK 1.8 uses: actions/setup-java@v3 with: distribution: 'zulu' java-version: 8 - name: Test with Maven env: nexus_username: ${{ secrets.nexus_username }} nexus_password: ${{ secrets.nexus_password }} KEY: ${{ secrets.decryption_key }} run: mvn test --settings .maven/settings.xml -Dsecure.key="$KEY" build: needs: test runs-on: ubuntu-latest steps: - name: Checkout this repo uses: actions/checkout@v3 - name: Cache dependencies uses: actions/cache@v3 with: path: ~/.m2/repository key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} restore-keys: | ${{ runner.os }}-maven- - name: Set up JDK 1.8 uses: actions/setup-java@v3 with: distribution: 'zulu' java-version: 8 - name: Build with Maven run: mvn -B package --file pom.xml -DskipMunitTests - name: Stamp artifact file name with commit hash run: | artifactName1=$(ls target/*.jar | head -1) commitHash=$(git rev-parse --short "$GITHUB_SHA") artifactName2=$(ls target/*.jar | head -1 | sed "s/.jar/-$commitHash.jar/g") mv $artifactName1 $artifactName2 - name: Upload artifact uses: actions/upload-artifact@v3 with: name: artifacts path: target/*.jar deploy: needs: build runs-on: ubuntu-latest steps: - name: Checkout this repo uses: actions/checkout@v3 - name: Cache dependencies uses: actions/cache@v3 with: path: ~/.m2/repository key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} restore-keys: | ${{ runner.os }}-maven- - uses: actions/download-artifact@v3 with: name: artifacts - name: Deploy to Sandbox env: USERNAME: ${{ secrets.anypoint_platform_username }} PASSWORD: ${{ secrets.anypoint_platform_password }} KEY: ${{ secrets.decryption_key }} run: | artifactName=$(ls *.jar | head -1) mvn deploy -DskipMunitTests -DmuleDeploy \ -Dmule.artifact=$artifactName \ -Danypoint.username="$USERNAME" \ -Danypoint.password="$PASSWORD" \ -Ddecryption.key="$KEY" ``` As you can see, we added a new `test` job that will run the MUnits before continuing with the rest of the jobs (build/deploy). We have seen the first steps before with the other two jobs, but the last step is different from what we’ve seen before. We also added the line `needs: test` under the build job so the jobs run synchronously. Finally, we added `-DskipMunitTests` to the two maven commands from build and deploy. Since we are running the tests on the first step, there’s no need to re-run them after that. The `nexus_username` and `nexus_password` are being passed as environment variables implicitly. While the `KEY` variable is being used in the maven command. Let’s take a closer look at the maven command. ```bash mvn test --settings .maven/settings.xml -Dsecure.key="$KEY" ``` Since we are using a specific configuration in our `settings.xml` to include the Nexus repository/credentials, we will be adding this file directly to the project (without hardcoding the credentials!!). We’ll see this configuration next. ⚠️ Another **important** change in this command is that this time we are sending the Mule property directly. In the previous post, we saw the decryption key property was called `secure.key`. If your property within the Mule project is named differently, please update this command appropriately. ## Set up your settings.xml As we just mentioned, we hadn’t created a `settings.xml` in the project before. Mainly because there was no need to do so. This time, since we’re using enterprise Nexus credentials to run the MUnit tests with Maven, we do need to set up specific credentials, profiles, and repositories. To start, create a new `.maven` folder at the root of your project. Inside this folder, create a `settings.xml` file. For example, this is how my folder structure looks like: ``` github-actions/ - .github/ - .maven/ - settings.xml - src/ - mule-artifact.json - pom.xml ``` Now copy and paste the following file. There’s no need to modify anything (just make sure the name is `settings.xml`). ```xml org.mule.tools MuleRepository ${nexus_username} ${nexus_password} Mule true MuleRepository MuleRepository https://repository.mulesoft.org/nexus-ee/content/repositories/releases-ee/ default true true ``` As you can see, the Nexus username and password credentials match the environment variables we are sending through the `build.yml` file. > [!DOCS] > For more information, see [Configuring Maven to Work with Mule](https://docs.mulesoft.com/mule-runtime/3.9/configuring-maven-to-work-with-mule-esb). ## Modify your pom.xml This time there’s nothing to add to your `pom.xml` file. The MUnit dependencies and needed configuration should be added automatically by MuleSoft once you add the tests to your code. However, here’s my project’s pom in case you want to compare it with yours: ```xml 4.0.0 com.mycompany github-actions 1.0.0-SNAPSHOT mule-application github-actions UTF-8 UTF-8 4.4.0 3.8.0 amartinez-github-actions Sandbox 2.3.13 org.apache.maven.plugins maven-clean-plugin 3.0.0 org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true https://anypoint.mulesoft.com ${app.runtime} ${anypoint.username} ${anypoint.password} ${app.name} ${env} MICRO us-east-2 1 true ${decryption.key} mule-application com.mulesoft.munit.tools munit-maven-plugin ${munit.version} test test test coverage-report true html org.mule.connectors mule-http-connector 1.6.0 mule-plugin org.mule.connectors mule-sockets-connector 1.2.2 mule-plugin com.mulesoft.modules mule-secure-configuration-property-module 1.2.5 mule-plugin com.mulesoft.munit munit-runner 2.3.13 mule-plugin test com.mulesoft.munit munit-tools 2.3.13 mule-plugin test org.mule.weave assertions 1.0.2 test anypoint-exchange-v3 Anypoint Exchange https://maven.anypoint.mulesoft.com/api/v3/maven default mulesoft-releases MuleSoft Releases Repository https://repository.mulesoft.org/releases/ default mulesoft-releases MuleSoft Releases Repository default https://repository.mulesoft.org/releases/ false ``` ## Run the pipeline That’s it! Once you’re done with the changes, simply push a new change to the **main** branch and this will trigger the pipeline. ![GitHub Actions run summary showing test, build, and deploy jobs all succeeding](../../assets/blog/part-3-ci-cd-pipeline-with-mulesoft-and-github-actions-munit-testing-4.png) ## More resources You can check out my [GitHub profile](https://github.com/alexandramartinez) for more CI/CD repos: - [github-actions](https://github.com/alexandramartinez/github-actions) to deploy a Mule app to CloudHub - [dataweave-utilities-library](https://github.com/alexandramartinez/dataweave-utilities-library) to publish a DataWeave library to Exchange - [api-catalog-cli-example](https://github.com/alexandramartinez/api-catalog-cli-example) to update APIs in Exchange using the API Catalog CLI I hope this was helpful! Don't forget to subscribe so you don't miss any future content. --- ## DataWeave programming challenge #3: Count palindrome phrases using the Strings module Source: https://prostdev.com/post/dataweave-programming-challenge-3 | Published: Mar 7, 2023 | Category: Challenges Try to solve this challenge on your own to maximize learning. We recommend you refer to the [DataWeave documentation](https://docs.mulesoft.com/dataweave/latest/) **only**. Try to avoid using Google or asking others so you can learn on your own and become a DataWeave expert! > [!PLAYGROUND] > [Solve on the Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=alexandramartinez%2Fdataweave-challenges&path=challenges%2F3) > [!TIP] > For this challenge, we encourage you to make use of the [Strings module](https://docs.mulesoft.com/dataweave/latest/dw-strings)'s functions. ## Input Consider the following input payload (can be of **txt** format): ``` this is not a palindrome blue Mr. Owl ate my metal worm Was it a car or a cat I saw? you're doing great! mug Rats live on no evil star what's taking so long? kayak Live on time, emit no evil Step on no pets boat Anna 2020/02/02 15/03/2001 01/02/2010 ``` ## Explanation of the problem Each line of the input payload is considered a different phrase. Review each phrase and find those that are [palindromes](https://en.wikipedia.org/wiki/Palindrome). Notice that punctuation, spaces, or special characters should not be considered to ensure a phrase is a palindrome. For example, - **Anna** in reverse is **anna** - this is a palindrome. - **2020/02/02** in reverse is **20200202** - this is a palindrome. - **Mr. Owl ate my metal worm** in reverse is **mrowlatemymetalworm** - this is a palindrome. After that, retrieve the character count of the **original** phrase. Including punctuation, spaces, and special characters. For example, - Anna is **4**. - 2020/02/02 is **10**. - Mr. Owl ate my metal worm is **25**. Return the total sum of all the palindrome's character count. ## Expected output In this case, the expected output would be: ``` 148 ``` The result for each of the phrases (whether they're palindrome or not) should be: - false - false - true - true - false - false - true - false - true - true - true - false - true - true - false - true ## Clues If you're stuck with your solution, feel free to check out some of these clues to give you ideas on how to solve it! ### Clue #1 Read the documentation of the [Strings module](https://docs.mulesoft.com/dataweave/latest/dw-strings) to make yourself familiar with the existing functions. ### Clue #2 If you've been following the previous challenges, we've learned that `payload splitBy "\n"` can be replaced with the `lines()` function from the Strings module. ### Clue #3 Use the lower() function to make sure all the characters can be compared in lower-case. ### Clue #4 Instead of using map() to iterate through the string's characters, you can use mapString() from the Strings module. ### Clue #5 You don't need to use regex to find the alphanumeric characters. You can use isAlphanumeric() from the Strings module. ### Clue #6 Instead of using selectors, you can use the reverse() function from the Strings module. ### Clue #7 Use sizeOf() and sum() to retrieve the character count in an array and then sum the complete array of numbers. ## Answer If you haven't solved this challenge yet, we encourage you to keep trying! It's ok if it's taking longer than you thought. We all have to start somewhere ✨ Check out the clues and read the docs before giving up. You got this!! 💙 There are many ways to solve this challenge, but you can find here one of our solutions so you can compare your result with us. ### Solution #1 ```dataweave %dw 2.0 import lines, isAlphanumeric, mapString, reverse from dw::core::Strings output application/json --- sum(lines(payload) map (line) -> do { var clearLine = lower(line) mapString ( if (isAlphanumeric($)) $ else "" ) var isPalindrome = clearLine == reverse(clearLine) --- if (isPalindrome) sizeOf(line) else 0 }) ``` Subscribe to receive notifications as soon as new content is published ✨ --- ## Part 2: CI/CD pipeline with MuleSoft and GitHub Actions - secured/encrypted properties Source: https://prostdev.com/post/part-2-ci-cd-pipeline-with-mulesoft-and-github-actions-secured-encrypted-properties | Published: Mar 2, 2023 | Category: Tutorials In the [previous article](https://www.prostdev.com/post/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions), we learned the basics to get started with a simple CI/CD pipeline to deploy a Mule application to CloudHub. We learned how to create Secrets in GitHub for your Anypoint Platform credentials, how to set up the `build.yml` file to set up the workflows, how to run the pipeline and the details of each of the steps in the pipeline. If you haven’t gone through that, I highly recommend you do. The steps for this post won’t be as detailed. In this post, we’ll see the steps to create a pipeline with GitHub Actions that will decrypt your secured properties in your Mule application. If you don’t have secured properties in your application, you may not need this configuration. ## Prerequisites You should already have all the setup in your Mule application for the secured properties. In summary, this is what you should already have: - Your secure properties file(s) under `src/main/resources` - Your encrypted properties (see [Encrypt Properties Using the Secure Properties Tool](https://docs.mulesoft.com/mule-runtime/latest/secure-configuration-properties#secure_props_tool)) - Your **Secure Properties Config** in Global Elements (`secure-properties:config`) - The name of your decryption key property (i.e., `secure.key`) - We recommend you add the key name to the `mule-artifact.json` file so it appears hidden in Runtime Manager (i.e., `"secureProperties": ["secure.key"]`) > [!IMPORTANT] > Make sure your Mule application is correctly configured and works locally before attempting to create the pipeline. Otherwise, you might encounter some issues. ## Set up your credentials In your GitHub repository, go to the **Settings** tab (make sure you are signed in to see it). Now go to **Secrets and variables > Actions**. Here you will be able to set up your repository secrets. In the previous tutorial, we added two credentials: - `ANYPOINT_PLATFORM_PASSWORD` - `ANYPOINT_PLATFORM_USERNAME` For this part, we’re going to add a new one that will contain the decryption key. Click on **New repository secret**. In the Name field, write `DECRYPTION_KEY`. In the Secret field, write the actual value of your key. For example, `MyMuleSoftKey`. Click **Add secret**. ![GitHub Actions New secret form with name DECRYPTION_KEY and value MyMuleSoftKey](../../assets/blog/part-2-ci-cd-pipeline-with-mulesoft-and-github-actions-secured-encrypted-properties-2.png) You don’t have to change the name of this secret. We will keep it as-is because it will be used in the pipeline. I’ll show you later in the post where to modify the name of the property to match the one you have in your Mule application (like `secure.key`). ## Set up your repo In the last post, we learned how to set up our `build.yml` file under `.github/workflows` for the actual pipelines in GitHub. This time, we are going to be using the same base file, just adding a few modifications to include the new decryption key. You can copy and paste the following code into this file. ```yaml name: Build and Deploy to Sandbox on: push: branches: [ main ] pull_request: branches: [ main ] jobs: build: runs-on: ubuntu-latest steps: - name: Checkout this repo uses: actions/checkout@v3 - name: Cache dependencies uses: actions/cache@v3 with: path: ~/.m2/repository key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} restore-keys: | ${{ runner.os }}-maven- - name: Set up JDK 1.8 uses: actions/setup-java@v3 with: distribution: 'zulu' java-version: 8 - name: Build with Maven run: mvn -B package --file pom.xml - name: Stamp artifact file name with commit hash run: | artifactName1=$(ls target/*.jar | head -1) commitHash=$(git rev-parse --short "$GITHUB_SHA") artifactName2=$(ls target/*.jar | head -1 | sed "s/.jar/-$commitHash.jar/g") mv $artifactName1 $artifactName2 - name: Upload artifact uses: actions/upload-artifact@v3 with: name: artifacts path: target/*.jar deploy: needs: build runs-on: ubuntu-latest steps: - name: Checkout this repo uses: actions/checkout@v3 - name: Cache dependencies uses: actions/cache@v3 with: path: ~/.m2/repository key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} restore-keys: | ${{ runner.os }}-maven- - uses: actions/download-artifact@v3 with: name: artifacts - name: Deploy to Sandbox env: USERNAME: ${{ secrets.anypoint_platform_username }} PASSWORD: ${{ secrets.anypoint_platform_password }} KEY: ${{ secrets.decryption_key }} run: | artifactName=$(ls *.jar | head -1) mvn deploy -DmuleDeploy \ -Dmule.artifact=$artifactName \ -Danypoint.username="$USERNAME" \ -Danypoint.password="$PASSWORD" \ -Ddecryption.key="$KEY" ``` As you can see, we added a new `KEY` variable that will get the decryption key from our secrets; and we are sending a `-Ddecryption.key` parameter into the maven command. > [!NOTE] > If you add this manually to the maven command, don’t forget to add a backslash (`\`) in the previous line. ## Modify your pom.xml Go to your Mule application’s `pom.xml` file and locate the `cloudHubDeployment` configuration. You will need to add the following property: > [!NOTE] > this is where the name of your actual property needs to be set. In our example, we are using `secure.key`. Modify this field to match the name of your property, but keep the `decryption.key` since we’re using that in the GitHub Action pipeline. ```xml ${decryption.key} ``` Your configuration should look something like this: ```xml org.apache.maven.plugins maven-clean-plugin 3.0.0 org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true https://anypoint.mulesoft.com ${app.runtime} ${anypoint.username} ${anypoint.password} ${app.name} ${env} MICRO us-east-2 1 true ${decryption.key} mule-application ``` That’s all! Once you send these changes to the main branch, your pipeline will start running and deploying to CloudHub. ## More resources You can check out my [GitHub profile](https://github.com/alexandramartinez) for more CI/CD repos: - [github-actions](https://github.com/alexandramartinez/github-actions) to deploy a Mule app to CloudHub - [dataweave-utilities-library](https://github.com/alexandramartinez/dataweave-utilities-library) to publish a DataWeave library to Exchange - [api-catalog-cli-example](https://github.com/alexandramartinez/api-catalog-cli-example) to update APIs in Exchange using the API Catalog CLI I hope this was helpful! Don't forget to subscribe so you don't miss any future content. --- ## DataWeave programming challenge #2: Rock Paper Scissors game score system Source: https://prostdev.com/post/dataweave-programming-challenge-2 | Published: Feb 28, 2023 | Category: Challenges > [!NOTE] > This challenge is based on [Advent of Code 2022 day 2](https://adventofcode.com/2022/day/2) Try to solve this challenge on your own to maximize learning. We recommend you refer to the [DataWeave documentation](https://docs.mulesoft.com/dataweave/latest/) **only**. Try to avoid using Google or asking others so you can learn on your own and become a DataWeave expert! > [!PLAYGROUND] > [Solve on the Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=alexandramartinez%2Fdataweave-challenges&path=challenges%2F2) ## Input Consider the following input payload (can be of **txt** format): ``` R R R P R S P R P P P S S R S P S S R R ``` ## Explanation of the problem Create a DataWeave script to keep your score on a series of Rock Paper Scissors games. The first column is what your **opponent** chose and the second column is what **you** chose. Each letter is a representation of the decision made: - R = Rock - P = Paper - S = Scissors The rules of the game are simple: - Rock defeats Scissors - Paper defeats Rock - Scissors defeat Paper As per the scoring system, you will have to keep track of whether you won, lost, or if it was a draw. Remember the second column is your choice. Here's how you will be counting the points: - 0 points if you lost the round - 3 points if the round was a draw - 6 points if you won the round For example, - The first round was **R R**, which means both your opponent and you chose Rock. This round results in a **draw**, so **3** points are added. - The second round was **R P**, which means your opponent chose Rock and you chose Paper. Paper defeats Rock, which means you win this round. **6** points are added. - The third round was **R S**, which means your opponent chose Rock and you chose Scissors. Rock defeats Scissors, which means you lose this round. **0** points are added. The final result will be the number of total points you earned in the 10 rounds. ## Expected output In this case, the expected output would be: ``` 30 ``` The result for each of the 10 rounds should be: - 3 - 6 - 0 - 0 - 3 - 6 - 6 - 0 - 3 - 3 ## Clues If you're stuck with your solution, feel free to check out some of these clues to give you ideas on how to solve it! ### Clue #1 Using `splitBy "\n"` is the same as using the lines() function from the Strings module ### Clue #2 You can use reduce() to keep the score in the accumulator ### Clue #3 If you don't want to use reduce, you can use map() to get the points of each round and add the numbers at the end ### Clue #4 You can use splitBy() to get your opponent's moves and yours to compare ### Clue #5 You can create a variable with the pointing system ### Clue #6 The rules can be made of objects in order to extract the points/moves ### Clue #7 If you used map() to get the points, you can use sum() to add all the numbers ### Clue #8 You can use the `do` operator to create localized variables ## Answer If you haven't solved this challenge yet, we encourage you to keep trying! It's ok if it's taking longer than you thought. We all have to start somewhere ✨ Check out the clues and read the docs before giving up. You got this!! 💙 There are many ways to solve this challenge, but you can find here some solutions we are providing so you can compare your result with us. ### Solution #1 - based on Felix Schnabel's solution ```dataweave %dw 2.0 output application/json import lines from dw::core::Strings var defeats = { R: "S", P: "R", S: "P" } fun getScore(opponent:String, me:String): Number = if (opponent == me) 3 else if (defeats[me] == opponent) 6 else 0 --- sum(lines(payload) map getScore($[0],$[-1])) ``` ### Solution #2 ```dataweave %dw 2.0 output application/json var rules = { R: { R: 3, P: 6, S: 0 }, P: { R: 0, P: 3, S: 6 }, S: { R: 6, P: 0, S: 3 } } --- payload splitBy "\n" reduce ((round, score=0) -> do { var arr = round splitBy " " var opponent = arr[0] var me = arr[-1] --- score + rules[opponent][me] }) ``` ### Solution #3 ```dataweave %dw 2.0 output application/json import lines from dw::core::Strings var rules = { R: { R: 3, P: 6, S: 0 }, P: { R: 0, P: 3, S: 6 }, S: { R: 6, P: 0, S: 3 } } --- lines(payload) map do { var arr = $ splitBy " " --- rules[arr[0]][arr[1]] } then sum($) ``` Subscribe to receive notifications as soon as new content is published ✨ --- ## How to develop a Battlesnake using a MuleSoft API and the DataWeave language Source: https://prostdev.com/post/how-to-develop-a-battlesnake-using-a-mulesoft-api-and-the-dataweave-language | Published: Feb 22, 2023 | Category: Tutorials If you’re not familiar with the modern Battlesnake game, you must at least remember the snake game from those old cell phones. It’s basically the same, but developing APIs and deploying them to the cloud...And fighting other snakes 😂 In this post, I’m going to guide you through all the steps to generate your own MuleSoft starter project in GitHub and be ready to start modifying your DataWeave code to move the snake. ## Create your own GitHub repository The first step is to get your own repo based on the Mule starter project that the great Manik created for us. To do this, you first have to go to the [official list of Starter Projects](https://docs.battlesnake.com/starter-projects) in the Battlesnake documentation. Scroll down until you see the **Mule and DataWeave Battlesnake Quickstart** project created by Manik Magar and click on it. ![Battlesnake starter projects grid with the Mule and DataWeave Quickstart highlighted](../../assets/blog/how-to-develop-a-battlesnake-using-a-mulesoft-api-and-the-dataweave-language-2.png) Once you open Manik’s GitHub repo, click on the **Use this template** button and select **Create a new repository**. ![GitHub repo with the Use this template menu open on Create a new repository](../../assets/blog/how-to-develop-a-battlesnake-using-a-mulesoft-api-and-the-dataweave-language-3.png) Add any name you choose for your repo and click **Create repository from template**. ![GitHub "Create a new repository from template" form with repo name my-snake](../../assets/blog/how-to-develop-a-battlesnake-using-a-mulesoft-api-and-the-dataweave-language-4.png) This will generate your own GitHub repo. You’re now ready to start modifying your code! ## Personalize your snake You can personalize your own snake. For example, changing its head/tail design or changing its color. You can modify this data at `src/main/resources/application.yaml`. This is what Manik added to the starter project: ```yaml battlesnake: config: "apiversion": "1" "author": "mmsonline" "color" : "#888888" "head" : "default" "tail" : "default" "version" : "0.0.1-beta" ``` And this is what I have for my snake: ```yaml battlesnake: config: "apiversion": "1" "author": "alexandramartinez" "color" : "#6c25be" "head" : "caffeine" "tail" : "nr-booster" "version" : "0.0.1-beta" ``` You can read more about the customizations in the [Battlesnake docs](https://docs.battlesnake.com/guides/customizations), or you can go ahead and search for the available [head/tail customizations](https://play.battlesnake.com/customizations). You can also leave the defaults as they are and your snake is still going to work. But it’s nice to be able to customize stuff :-) ## Find the DataWeave logic The API is not the problem here. Manik generated a pretty good starter project. The thing is that you need to create all the DataWeave code on your own to tell the snake where to move to. Make sure you read the [Battlesnake docs](https://docs.battlesnake.com/guides) so you can understand how everything works. But I’ll give you a quick summary. You will receive a [POST /move](https://docs.battlesnake.com/api/requests/move) request in your API containing the current turn. This will trigger the DataWeave code (located at `src/main/resources/dw/move-snake.dwl`) and return a response of where to move from here. All your DataWeave code will return is a JSON object telling the game where you’re gonna move next. For example: ```json { "move": "up", "shout": "Moving up!" } ``` The trick here is to know exactly where to move based on a series of rules. You will receive a payload with a general structure that looks something like this: ```json { "game": { "id": "...", "ruleset": { "name": "standard", "settings": {...} } }, ... }, "turn": 42, "board": { "height": 11, "width": 11, "snakes": [ { "id": "...", "name": "...", "body": [...], "head": {...}, ... } "food": [...], "hazards": [] }, "you": { "body": [...], "head": {...}, "length": 3, ... } } ``` > To see a full example, please refer to the docs [here](https://docs.battlesnake.com/api/example-move#move-api-request). So, your code will have to make decisions based on where the other snakes are, where the walls are, where the food is, etc. This is the challenging part! Manik gives you a pretty good way to start coding the logic. But the starter snake only knows to not collide with its own neck. It doesn’t know how to not hit walls, not hit itself, not hit others, or find food. You will need to code that on your own. ## Deploy the Mule app Whenever you feel ready to try out your snake’s logic, you can deploy your Mule application directly to **CloudHub** and take the URL from there. It should look like this: ``` http://amartinez-mule-battlesnake.us-e2.cloudhub.io ``` > For information on how to deploy your application, see [Deploy to CloudHub](https://docs.mulesoft.com/cloudhub-1/deploying-to-cloudhub). > [!TIP] > If your Anypoint Platform account expires, you can create a new account using the same details as before (name, email, phone, etc.) and just changing your username. However, deploying to CloudHub can take a long time. I’ve personally been using **ngrok** to do local deployments and be able to expose my local URL through the public internet. You can follow the next steps to use ngrok: - Install [ngrok](https://ngrok.com/) - Run your Mule application locally - Run `ngrok http 8081` - Retrieve the public URL This URL would look something similar to this: ``` https://c0cb-2001-1970-5425-e000-00-7e86.ngrok.io ``` Now you’re all ready to create your actual snake on Battlesnake! ## Create your Battlesnake First of all, you need to create an account at [play.battlesnake.com](https://play.battlesnake.com/). Once you have it, you can go to the **Battlesnakes** tab, click on **New Battlesnake**, and fill out the details. Make sure you add `/api` to the end of the URL (either CloudHub or ngrok). Here’s my snake if you want to take it as an example: ![Battlesnake settings for "Maxine the Mule" with the CloudHub API URL and tags](../../assets/blog/how-to-develop-a-battlesnake-using-a-mulesoft-api-and-the-dataweave-language-5.png) After you click on **Save**, it will attempt to call the URL. If the app was successfully deployed, there shouldn’t be any issue with saving it. Now you’re ready to test your snake! ## Create a new game Go to the **Create Game** tab. I recommend you leave the default options selected. Scroll down if needed and click **Add to Game** next to your snake. ![Battlesnake Create Game screen with Standard map and My Battlesnakes listed](../../assets/blog/how-to-develop-a-battlesnake-using-a-mulesoft-api-and-the-dataweave-language-6.png) You can also click on the **Official Bots** tab to select more snakes for the game or click on **Public Battlesnakes** to add your friends’ snakes. ![Create Game screen with Official Bots added alongside Maxine the Mule, ready to start](../../assets/blog/how-to-develop-a-battlesnake-using-a-mulesoft-api-and-the-dataweave-language-7.png) Once you feel the game is ready, simply click on **Start Game** and then press the play (▶️) button. ![Animated Battlesnake game board with snakes moving toward food pellets](../../assets/blog/how-to-develop-a-battlesnake-using-a-mulesoft-api-and-the-dataweave-language-8.gif) Look at my Maxine go! ## Add your snake to the leaderboard Once you have a good and stable snake, you can go to the [leaderboards](https://play.battlesnake.com/leaderboards) and add your Battlesnake so it can start earning points and get a place! These are my stats so far: ![Alex Martinez's Battlesnake profile showing Standard and Duels ratings and ranks](../../assets/blog/how-to-develop-a-battlesnake-using-a-mulesoft-api-and-the-dataweave-language-9.png) I work on my snake at least once per week during my live streams. You can watch me live on [Twitch](https://www.twitch.tv/devalexmartinez) or watch my previous streams on [YouTube](https://www.youtube.com/playlist?list=PLb61lESgk6hi60IazebMZ7pmZBZeIcEKt). I hope this was helpful! Feel free to play around with Maxine the Mule if you want :) just search for/select her from the **Public Battlesnakes**. Send me any good games you have with her! I’d love to see them playing :D --- ## DataWeave programming challenge #1: Add numbers separated by paragraphs and get the max number Source: https://prostdev.com/post/dataweave-programming-challenge-1 | Published: Feb 21, 2023 | Category: Challenges > [!NOTE] > This challenge is based on [Advent of Code 2022 day 1](https://adventofcode.com/2022/day/1) Try to solve this challenge on your own to maximize learning. We recommend you refer to the [DataWeave documentation](https://docs.mulesoft.com/dataweave/latest/) **only**. Try to avoid using Google or asking others so you can learn on your own and become a DataWeave expert! > [!PLAYGROUND] > [Solve on the Playground](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=alexandramartinez%2Fdataweave-challenges&path=challenges%2F1) ## Input Consider the following input payload (can be of **txt** format): ``` 1 2 3 4 5 0 15 20 5 1 3 25 6 4 4 ``` ## Explanation of the problem Create a DataWeave script to add all the numbers separated by a paragraph. For example, - the first number would be **3** (2 + 1), - the second number would be **12** (3 + 4 + 5 + 0), - the third number would be **15**, and so on. After that, retrieve the highest number. *(Using code, not manually* 😉) ## Expected output In this case, the expected output would be: ``` 29 ``` ## Clues If you're stuck with your solution, feel free to check out some of these clues to give you ideas on how to solve it! ### Clue #1 Use the splitBy() function ### Clue #2 Instead of splitting by "\n" try splitting by "\n\n" at first - to separate by paragraph ### Clue #3 Use map() to go through each paragraph ### Clue #4 Use splitBy() again to separate each of the paragraph's lines ### Clue #5 Use sum() to add all the numbers in each paragraph ### Clue #6 Use max() to return the highest number from the array ### Clue #7 If some of the numbers are string instead of numbers, use `as` to coerce them to Numbers -- this is a bug and will hopefully be fixed soon! ## Answer If you haven't solved this challenge yet, we encourage you to keep trying! It's ok if it's taking longer than you thought. We all have to start somewhere ✨ Check out the clues and read the docs before giving up. You got this!! 💙 There are many ways to solve this challenge, but you can find here some solutions we are providing so you can compare your result with us. ### Solution #1 ```dataweave %dw 2.0 output application/json --- (payload splitBy "\n\n") map (sum($ splitBy "\n") as Number) then max($) ``` ### Solution #2 ```dataweave %dw 2.0 output application/json --- (payload splitBy "\n\n") map ((item) -> ( sum(item splitBy "\n") as Number )) then max($) ``` ### Solution #3 ```dataweave %dw 2.0 output application/json --- (payload splitBy "\n\n") map ((item,index) -> do { var newItem = item splitBy "\n" --- sum(newItem) as Number }) then max($) ``` ### Solution #4 ```dataweave %dw 2.0 output application/json --- max((payload splitBy "\n\n") map (sum($ splitBy "\n") as Number)) ``` Did you like this challenge?? Let us know what you think!! 👇 Subscribe to receive notifications as soon as new content is published ✨ --- ## DataWeave 2.0 scopes for local variables: 'using' vs. 'do' operators Source: https://prostdev.com/post/dataweave-2-0-scopes-for-local-variables-using-vs-do-operators | Published: Feb 14, 2023 | Category: Guides I had previously written a post with [my top 5 DataWeave tips to make your life easier](https://www.prostdev.com/post/my-top-5-dataweave-tips-to-make-your-life-easier) where I talked about the *do* operator. However, I’ve seen a lot of people asking about the *using* operator since it was inherited from DataWeave 1.0. In this post, I’ll go through some of the main differences between these two operators so you decide which one to use in your DataWeave scripts! > [!NOTE] > By this point, I’m assuming you already have the basic DataWeave understanding. If this is not the case, I recommend you read [these tutorials first](https://developer.mulesoft.com/tutorials-and-howtos/dataweave/what-is-dataweave-getting-started-tutorial/). ## What are scopes? If you come from a different programming paradigm, this concept might be easy to understand. To keep it simple, let’s just say that scopes are these imaginary boxes where you can keep variables, functions, and so on. These variables (or functions) are not available to anyone else outside of these boxes, only inside. We will see some examples in a bit. In DataWeave, you can have **global** and **local** variables (and functions, annotations, etc.). The global variables are the ones that appear over the three dashes (---) that separate your script. For example ```dataweave %dw 2.0 output application/json var globalVar = "this is my global variable" --- globalVar ``` These variables are accessible from anywhere in your script. They don’t have to exist inside a specific scope to be used. Let’s talk about how to create local variables with the *using* or *do* operators so we can put together the meaning of *scopes*. ## Creating scopes and local variables with the *do* operator Try out the following example. What age do you think will be outputted from this script? ```dataweave %dw 2.0 output application/json var age = 21 --- { person: do { var user = "Robin" var age = 5 --- { name: user, age: age } } } ``` The answer is 5. We used the *do* operator in line 6 to create a local scope. Notice how we created the variables *user* and *age* inside this scope. This new scope will work very similarly to a regular script in the sense that it can have its variables, functions, etc. Notice how we have to add three dashes (---) in line 9 to separate the variable declaration from the localized script. This is because when you use the *do* operator, you can use this syntax for your new local scope. Below is a graphical representation of how these two scopes are separated. ![DataWeave script with the global scope outlined in green and the do-operator local scope in red](../../assets/blog/dataweave-2-0-scopes-for-local-variables-using-vs-do-operators-2.png) The age in the output is 5 because it will take the “closest” variable. In other words, the more local the variable, the more precedence it takes. What do you think will be the output if you use the following code? Try to figure it out! The answer will be at the end of the post 🙂 ```dataweave %dw 2.0 output application/json var age = 21 --- { person: do { var user = "Robin" var age = 5 --- { name: user, age: age + do { var age = 10 --- age } } } } ``` You can create as many local scopes as you want, but I wouldn’t recommend having more than 1 unless completely necessary. Mainly to avoid spaghetti code. You can read more about *do* in this post: [My top 5 DataWeave tips to make your life easier](https://www.prostdev.com/post/my-top-5-dataweave-tips-to-make-your-life-easier). ## Creating local variables with the *using* operator Now, I have to be completely honest here. I haven’t used the *using* operator. I did work on DataWeave 1.0 for a while, but it was confusing for me to understand how this syntax worked, so I never really used it. Here is an awesome post if you’re still developing with DataWeave 1.0: [How to use the “Using” Operator in DataWeave, the MuleSoft Mapping Tool](https://bitsinglass.com/mulesoft-dataweave-operator-tips/). Otherwise, I’ll explain how to create local variables in DataWeave 2.0 here. Let’s follow the previous example to demonstrate. ```dataweave %dw 2.0 output application/json var age = 21 --- { person: using (user = "Robin", age = "5") { name: user, age: age } } ``` We will have the same functionality as before but see how this syntax is different than *do*. We can also create variables with lambda expressions to use them as functions. Like in the following example. ```dataweave %dw 2.0 output application/json var age = 21 --- { person: using ( user = "Robin", age = "5", echo = (t) -> t ) { name: user, age: age, test: echo(123) } } ``` However, this operator is considered deprecated since DataWeave 2.0. I personally don’t like the way the syntax works. It makes more sense for me to use *do* instead. But you can still use *using* if you’re really into it 😀- although you might get a huge warning in your scripts telling you to stop using this operator! ![Anypoint Studio warning that the using operator is deprecated, suggesting replacing it with do](../../assets/blog/dataweave-2-0-scopes-for-local-variables-using-vs-do-operators-3.png) ## My advice for you The *using* operator is deprecated for a reason. The fact that the team didn’t remove it from DataWeave 2.0 doesn’t mean that it’s the best way to create local variables. Yes, you can still use it if that’s the way you’ve been doing it since DataWeave 1.0, but who knows when they’ll remove it? It might be risky to keep it that way. I also feel it’s more intuitive to use *do* because of the three-dashes syntax that we already know how to use. We can easily declare local variables, functions, annotations, or namespaces the same way we do global ones. However, it can be tricky to use at first because of the curly brackets ( { } ). But I swear, the more you use it, the less time you take to do it automatically! You can also check out the docs to read more about these two: - (DataWeave 2.0) [Scope and Flow Control Operators](https://docs.mulesoft.com/dataweave/latest/dw-operators#scope-and-flow-control-operators) - (DataWeave 2.0) [Example: Local DataWeave Variables](https://docs.mulesoft.com/dataweave/latest/dataweave-variables#example_local_variable) - (DataWeave 1.0) [Scoped Variables](https://docs.mulesoft.com/dataweave/1.2/dataweave-reference-documentation#scoped-variables) Oh! And the answer to the question is this: ```json { "person": { "name": "Robin", "age": 15 } } ``` Try it out if you don’t believe me :-) Let me know if you have any questions! -alex --- ## How to generate shareable link examples from GitHub to open in the DataWeave Playground Source: https://prostdev.com/post/how-to-generate-shareable-link-examples-from-github-to-open-in-the-dataweave-playground | Published: Feb 7, 2023 | Category: Tutorials Some months ago, the DataWeave engineering team created this beautiful functionality to be able to open DataWeave examples (including inputs and scripts) directly in the Playground with a single link. [Here’s an example.](https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=alexandramartinez%2Fdataweave-scripts&path=functions%2FmaskFields) In this post, we’re going to learn how to create these shareable links from a GitHub repository to open directly in the online DataWeave Playground. ## The link's anatomy Let’s use as an example the link I shared at the beginning of the post. This is the complete URL: ``` https://dataweave.mulesoft.com/learn/playground?projectMethod=GHRepo&repo=alexandramartinez%2Fdataweave-scripts&path=functions%2FmaskFields ``` Now let’s separate this to make it more readable: | URL part | Description | |---|---| | https://dataweave.mulesoft.com/learn/playground | The link should always start with this. This is the URL to open the Playground. | | ? | Marks the start of the query parameters in this URI. | | & | Marks the concatenation of more query parameters. | | `projectMethod`=GHRepo | This query parameter should always stay the same. | | `repo`=alexandramartinez%2Fdataweave-scripts | This query parameter indicates where the GitHub repository is located.
This has to be a **public** repo.
In this case, it’s referencing this repo: [github.com/alexandramartinez/dataweave-scripts](https://github.com/alexandramartinez/dataweave-scripts) | | `path`=functions%2FmaskFields | This query parameter indicates the path inside the repo where the inputs and scripts are located.
In this case, it’s referencing this folder: [alexandramartinez/dataweave-scripts/tree/main/functions/maskFields](http://github.com/alexandramartinez/dataweave-scripts/tree/main/functions/maskFields) | You may have noticed that there are some weird characters in the query parameters’ values (%2F). This is just the representation of a slash (/) to make it safe to pass it down in a URL. To make sure your values for the *repo* and *path* query parameters are URL-safe, you can use this tool: [urlencoder.org](https://www.urlencoder.org/). Simply input your value at the top and click Encode. The encoded value will be returned under the button. Now that we have a better understanding of how the link is built, let’s now make sure our folder structure inside our repo (the *maskFields* folder from the previous example) is correctly set. ## The folder structure Since we are only passing the main folder to the Playground, we have to make sure everything is where it’s supposed to be. For that, we have to set the correct folder structure. The folder structure from the previous [maskFields](https://github.com/alexandramartinez/dataweave-scripts/tree/main/functions/maskFields)[folder](https://github.com/alexandramartinez/dataweave-scripts/tree/main/functions/maskFields) example would look like this: ``` maskFields/ - inputs/ - payload.xml - transform.dwl ``` As you can see, this is a simple example where there’s only one payload and only one main script (`transform.dwl`). Let’s see a more detailed explanation of what this all means. | **Folder/file** | **Description** | [Playground section](https://developer.mulesoft.com/tutorials-and-howtos/dataweave/learn-dataweave-with-the-dataweave-playground-getting-started/) | |---|---|---| | inputs/ | This folder should always be named ‘inputs’. This is where you can add several inputs if needed.
Make sure the files have the appropriate extension (.xml, .json, .txt) for the Playground to know which data format it is.
If you’re adding several payloads (for example, one JSON and one XML), make sure they are named differently. You reference these payloads from your scripts using the file name without the extension. | Input Explorer | | `transform.dwl` | This is the main file where your DataWeave script is located. The file name should always be `transform.dwl`. | Script Explorer / Script | | Additional modules | If you want to add custom DataWeave modules to the repo, you should do it at the same level as the `transform.dwl` file. These modules can have different names but they should all have the .dwl extension. | Script Explorer / Script | Once you have everything set up and committed/pushed to your **public** GitHub repository, you will be able to use the link to share your examples :) > [!NOTE] > It may take up to 30 minutes for the Playground to be updated after you make any changes to the repository. So, if you do not see your changes being reflected immediately, just wait a bit. Feel free to contact me for any questions or troubleshooting steps regarding this. I’m happy to help! -alex --- ## Simplified try-catch strategy in DataWeave with the default keyword (instead of try/orElse) Source: https://prostdev.com/post/simplified-try-catch-strategy-in-dataweave-with-the-default-keyword-instead-of-try-orelse | Published: Jan 31, 2023 | Category: Guides The other day I was reading some StackOverflow DataWeave questions (as one does) and I got to this very interesting thread: [Why the “default” keyword acts like “try + catch / orElse” in some cases](https://stackoverflow.com/questions/73785223/why-the-default-keyword-acts-like-try-catch-orelse-in-some-cases). I wanted to take a moment to create an article about my own explanation of what I see in that thread. I already created a video about it, so, I might as well create a blog post too :) P.S. You can try out the scripts in the [DataWeave Playground](https://dataweave.mulesoft.com/learn/playground) or in [Visual Studio Code](https://www.prostdev.com/post/how-to-move-your-code-from-the-dataweave-playground-to-visual-studio-code). ## The question In summary, [Harshank Bansal](https://stackoverflow.com/users/10946202/harshank-bansal) notices that if you try to run the following script, you will receive an error message stating that *ABC cannot be coerced to Number* (which is expected). Script: ```dataweave %dw 2.0 output application/json --- "ABC" as Number ``` Output: ``` Cannot coerce String (ABC) to Number 4| "ABC" as Number ^^^^^^^^^^^^^^^ Trace: at main::main (line: 4, column: 1) ``` Maybe your first instinct to try to fix this code - if you’re familiar with the [try](https://docs.mulesoft.com/dataweave/latest/dw-runtime-functions-try) function - would be to do a script similar to the one Harshank mentions in the question: ```dataweave %dw 2.0 import * from dw::Runtime output application/json --- try(() -> ("ABC" as Number)) orElse "Invalid number" ``` This correctly catches the error and outputs the “Invalid number” string instead. However, Harshank found an alternative for this specific use case which is using the [default](https://docs.mulesoft.com/dataweave/latest/dataweave-cookbook-defaults) keyword. Like so: ```dataweave %dw 2.0 output application/json --- ("ABC" as Number) default "Invalid number" ``` This previous script outputs the exact same “Invalid number” string as we did with the *try* function. However, this seems a bit weird since the *default* keyword is mainly used for *null* values and this is an error. Now let’s talk about the answer/explanation that Mariano de Achaval wrote on why this behavior happens. ## The answer If you don’t know who [Mariano](https://stackoverflow.com/users/1472690/machaval) is, he’s one of the architects that started developing DataWeave years ago. That’s why his answer is a big deal :) Mariano mentions that the *default* keyword can be used for two things: - When the left-side value is *null*, the right-side value is returned. - To handle **some** exceptions like the coercion one we saw at the beginning of the post. Notice how I wrote **some** in bold there, that’s because Mariano and Harshank both mention it doesn’t work for all possible exceptions. It only works when: - You call a function that doesn’t support *null*. - You try to coerce a value to a data type (like the example we saw). Now, it is worth mentioning that this behavior is available to use but it is not recommended (as Mariano wrote). It is better to use the *try* function instead. ## My takeaway Even though it’s not recommended to use, it doesn’t hurt to know it exists :) Sometimes you need to do a quick script where you need to ensure the type coercion doesn’t return an exception (like Harshank’s example) and the code isn’t really too complex to add the *try* function. What is **your** takeaway from this?? -alex --- ## Part 1: How to set up a CI/CD pipeline to deploy your MuleSoft apps to CloudHub using GitHub Actions Source: https://prostdev.com/post/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions | Published: Jan 24, 2023 | Category: Tutorials Creating CI/CD (Continuous Integration, Continuous Delivery) pipelines for your code has become a standard practice. Instead of worrying about deployments and keeping your environments up to date, you can set up this automated pipeline to do the deployments for you. You can focus on developing your code and let the pipeline take care of the rest. In this post, we’ll learn how to create a simple CI/CD pipeline to deploy Mule applications from GitHub to CloudHub using GitHub Actions. We’ll be using [this GitHub repo](https://github.com/alexandramartinez/github-actions) for this example. ## Set up your repo We won’t go into the details of creating a GitHub repo with your Mule application. You can take a look at the [example repo](https://github.com/alexandramartinez/github-actions) we’ll be using throughout the post so we can focus on explaining the CI/CD setup. Once you have your Mule application in a GitHub repo, create a folder called `.github`, and inside it, create another folder called `workflows`. Here, create a new `build.yml` file and paste the following contents into it. ```yaml name: Build and Deploy to Sandbox on: push: branches: [ main ] pull_request: branches: [ main ] jobs: build: runs-on: ubuntu-latest steps: - name: Checkout this repo uses: actions/checkout@v3 - name: Cache dependencies uses: actions/cache@v3 with: path: ~/.m2/repository key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} restore-keys: | ${{ runner.os }}-maven- - name: Set up JDK 1.8 uses: actions/setup-java@v3 with: distribution: 'zulu' java-version: 8 - name: Build with Maven run: mvn -B package --file pom.xml - name: Stamp artifact file name with commit hash run: | artifactName1=$(ls target/*.jar | head -1) commitHash=$(git rev-parse --short "$GITHUB_SHA") artifactName2=$(ls target/*.jar | head -1 | sed "s/.jar/-$commitHash.jar/g") mv $artifactName1 $artifactName2 - name: Upload artifact uses: actions/upload-artifact@v3 with: name: artifacts path: target/*.jar deploy: needs: build runs-on: ubuntu-latest steps: - name: Checkout this repo uses: actions/checkout@v3 - name: Cache dependencies uses: actions/cache@v3 with: path: ~/.m2/repository key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} restore-keys: | ${{ runner.os }}-maven- - uses: actions/download-artifact@v3 with: name: artifacts - name: Deploy to Sandbox env: USERNAME: ${{ secrets.anypoint_platform_username }} PASSWORD: ${{ secrets.anypoint_platform_password }} run: | artifactName=$(ls *.jar | head -1) mvn deploy -DmuleDeploy \ -Dmule.artifact=$artifactName \ -Danypoint.username="$USERNAME" \ -Danypoint.password="$PASSWORD" ``` > [!NOTE] > You can name this YAML file however you want. It doesn’t have to be `build`. ![GitHub viewing the build.yml workflow file at .github/workflows on the main branch.](../../assets/blog/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions-2.png) This file is where the CI/CD pipeline is set up. Before we get into the details of this file, let’s first set up the Actions secrets with your Anypoint Platform credentials. ## Set up your credentials In your GitHub repository, go to the **Settings** tab (make sure you are signed in to see it). Now go to **Secrets and variables** **>** **Actions**. Here you will be able to set up your repository secrets. ![GitHub repo Actions secrets page listing the ANYPOINT_PLATFORM_PASSWORD and ANYPOINT_PLATFORM_USERNAME secrets.](../../assets/blog/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions-3.png) Click on **New repository secret**. In the **Name** field, write `ANYPOINT_PLATFORM_PASSWORD`. In the **Secret** field, write the actual Anypoint Platform password you use to sign in to Anypoint Platform. Click **Add secret**. ![New secret form with Name ANYPOINT_PLATFORM_PASSWORD and a password value, with the Add secret button.](../../assets/blog/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions-4.png) Repeat the same process for the `ANYPOINT_PLATFORM_USERNAME` secret. Add here the username you use to sign in to Anypoint Platform. ![New secret form with Name ANYPOINT_PLATFORM_USERNAME and a username value, with the Add secret button.](../../assets/blog/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions-5.png) That’s it for the basic setup! Let’s take a closer look into the `build.yml` file to make sure everything is set up properly for your case. ## Inside the build.yml file Let’s go through the contents of the `build.yml` file so you can personalize it for your own needs. If you want to skip this explanation, you can go straight to Run the pipeline. ### name The first thing we see is a `name` field. This is the name of the Action that will appear in the UI under the **Actions** tab. You can name this anything you want. ```yaml name: Build and Deploy to Sandbox ``` ![GitHub Actions tab showing the "Build and Deploy to Sandbox" workflow with its list of runs.](../../assets/blog/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions-6.png) ### on Next, we have the `on` field. This tells you when the Action will run. In this case, we have it set up to run anytime there’s a push to the **main** branch, or when a Pull Request has been merged into the **main** branch. You can also set it up to run with a schedule, but we won’t go through that for this example. You can see a schedule example [here](https://github.com/alexandramartinez/alexandramartinez/blob/main/.github/workflows/blog-post-workflow.yml). ```yaml on: push: branches: [ main ] pull_request: branches: [ main ] ``` ### jobs Next, we have the `jobs` field. Technically, you can just create a huge job and run all the steps inside it, but it will be overwhelming for you if you need to troubleshoot a step. It is best to split it up into different jobs when possible. It’s worth noting that jobs run async to each other. We’ll learn how to create async jobs when we see the `deploy` job in a moment. In our case, we have two jobs: `build` and `deploy`. ```yaml jobs: build: [...] deploy: [...] ``` ![Successful workflow run summary with the build and deploy jobs both passing in 3m 9s.](../../assets/blog/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions-7.png) ### jobs: build Inside the `build` job, we start by stating the operating system in which it will run. In this case, ubuntu. ```yaml runs-on: ubuntu-latest ``` Then, we start listing the `steps` for the job. The first two steps are repeated in both jobs. First, we have to check out the current repository in the ubuntu machine where the pipeline is running. ```yaml - name: Checkout this repo uses: actions/checkout@v3 ``` After that, we set up a cache for the Maven dependencies in the ubuntu machine’s `.m2` folder. ```yaml - name: Cache dependencies uses: actions/cache@v3 with: path: ~/.m2/repository key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} restore-keys: | ${{ runner.os }}-maven- ``` Now, for this `build` job, we need to set up Java 8 in order to run the Maven command to build the application into a jar file. ```yaml - name: Set up JDK 1.8 uses: actions/setup-java@v3 with: distribution: 'zulu' java-version: 8 ``` After setting up Java 8, we can now build this Mule application with Maven. ```yaml - name: Build with Maven run: mvn -B package --file pom.xml ``` Next, for troubleshooting purposes, we’ll change the name of the generated jar file to include the last commit’s hash. This file name will be visible in CloudHub so you can make sure the application deployed is the correct version. ```yaml - name: Stamp artifact file name with commit hash run: | artifactName1=$(ls target/*.jar | head -1) commitHash=$(git rev-parse --short "$GITHUB_SHA") artifactName2=$(ls target/*.jar | head -1 | sed "s/.jar/-$commitHash.jar/g") mv $artifactName1 $artifactName2 ``` Finally, we will upload this artifact (the jar file) so it becomes available for the next job. ```yaml - name: Upload artifact uses: actions/upload-artifact@v3 with: name: artifacts path: target/*.jar ``` ### jobs: deploy We had mentioned earlier that jobs run asynchronously to each other but there’s a way to configure them to run synchronously. In this case, we need to run the `build` job before the `deploy` job. Otherwise, we won’t have any jar file to deploy to CloudHub. To do this, we use the `needs` field inside the `deploy` job to make sure the `build` job runs before this one. ```yaml needs: build ``` After that, the `runs-on` and the first two steps are the same as the previous job. The third step, however, is not to set up Java 8 because we no longer need to build the Mule application. Now, we need to download the artifact (jar file) we uploaded in the last step of the previous job. ```yaml - uses: actions/download-artifact@v3 with: name: artifacts ``` After we download it, we can now deploy it to CloudHub using Maven. This next step will take the two Action secrets we had previously set up in the Set up your credentials section: `ANYPOINT_PLATFORM_PASSWORD` and `ANYPOINT_PLATFORM_USERNAME`. ```yaml - name: Deploy to Sandbox env: USERNAME: ${{ secrets.anypoint_platform_username }} PASSWORD: ${{ secrets.anypoint_platform_password }} #DECRYPTION_KEY: ${{ secrets.decryption_key }} run: | artifactName=$(ls *.jar | head -1) mvn deploy -DmuleDeploy \ -Dmule.artifact=$artifactName \ -Danypoint.username="$USERNAME" \ -Danypoint.password="$PASSWORD" # -Ddecryption.key="$DECRYPTION_KEY" ``` You will notice that there are two commented-out lines here. If your Mule application uses any sort of key to decrypt secured properties, this is where you can add it. Just make sure you create a `DECRYPTION_KEY` secret too. ## Run the pipeline Since we set up our pipeline to run every time there’s a new commit into the **main** branch, we can simply make a change to the current Mule application and commit it into the branch. Once this is done, the pipeline will be triggered. You can go into the **Actions** tab to see the workflows. ![GitHub Actions tab showing the "Build and Deploy to Sandbox" workflow with its list of runs.](../../assets/blog/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions-6.png) If you want to see a specific run, you can click on the title of the commit (for example, **Update `README.md`**). This will show the specifics of the pipeline run, like the jobs it ran and whether they were successful or not. ![Successful workflow run summary with the build and deploy jobs both passing in 3m 9s.](../../assets/blog/how-to-set-up-a-ci-cd-pipeline-to-deploy-your-mulesoft-apps-to-cloudhub-using-github-actions-7.png) You can also click on each specific job to get a closer look at the steps and the full logs of each step. This is helpful to troubleshoot when something goes wrong in the run. ## More resources You can check out my [GitHub profile](https://github.com/alexandramartinez) for more CI/CD repos: - [github-actions](https://github.com/alexandramartinez/github-actions) to deploy a Mule app to CloudHub - [dataweave-utilities-library](https://github.com/alexandramartinez/dataweave-utilities-library) to publish a DataWeave library to Exchange - [api-catalog-cli-example](https://github.com/alexandramartinez/api-catalog-cli-example) to update APIs in Exchange using the API Catalog CLI If you want more details about CI/CD and how I got to create this repo with these settings, you can check out the [CI/CD collection](https://www.twitch.tv/collections/uMqro6u7JRcMWQ) from our Twitch channel. > [!NOTE] > The initial versions of the pipeline are based on the following repository created by Archana Patel: [arch-jn/github-actions-mule-cicd-demo](https://github.com/arch-jn/github-actions-mule-cicd-demo). I hope this was helpful! Don't forget to subscribe so you don't miss any future content. -alex --- ## How to move your code from the DataWeave Playground to Visual Studio Code Source: https://prostdev.com/post/how-to-move-your-code-from-the-dataweave-playground-to-visual-studio-code | Published: Jan 17, 2023 | Category: Tutorials ![Photo by Safar Safarov on Unsplash.](../../assets/blog/how-to-move-your-code-from-the-dataweave-playground-to-visual-studio-code-1.png) If you’re not familiar with the online DataWeave Playground yet, this is an online tool that you can access from your browser to develop DataWeave code without having to open Anypoint Studio. It is mainly recommended for training purposes because of its limitations - since it’s running from your browser, it’s limited to the amount of memory allocated in your browser. There is a workaround to run it locally with the Docker image (see [How to run locally the DataWeave Playground Docker Image](https://www.prostdev.com/post/how-to-run-locally-the-dataweave-playground-docker-image)) but this image might stop being supported in the future. That’s why it’s best to start using Visual Studio Code if you want to develop more complex code and need more performance for bigger payloads. Here’s a video to complement your learnings from this article: ## Advantages Here are a few advantages of using the DataWeave extension for Visual Studio Code instead of the DataWeave Playground (there are more, but these seemed more important for now): - Handle bigger payloads - Get code suggestions - Create unit tests for your code - Create documentation for your code - Debug your scripts ## Install the extension on VSCode Before starting, make sure your extension on VSCode has been correctly installed. You can follow the next steps or see a detailed guide here: [Getting started with the DataWeave extension for Visual Studio Code](https://developer.mulesoft.com/tutorials-and-howtos/dataweave/dataweave-extension-vscode-getting-started/). - Install [Visual Studio Code](https://code.visualstudio.com/Download). - Install [Java 8](https://www.oracle.com/ca-en/java/technologies/javase/javase8-archive-downloads.html) and [Maven](https://maven.apache.org/download.cgi) version 3.6.3 or higher. - Install the [DataWeave extension](https://marketplace.visualstudio.com/items?itemName=MuleSoftInc.dataweave) for Visual Studio Code. And that’s it! ## Export the code from the Playground We are assuming you already have some code going on in the DataWeave Playground. If this is not the case, you can simply [open the Playground](https://dataweave.mulesoft.com/learn/playground) and use the sample code that is automatically generated. 1. From the DataWeave Playground, click on the **Export** button at the top. ![DataWeave Playground Export and Import buttons](../../assets/blog/how-to-move-your-code-from-the-dataweave-playground-to-visual-studio-code-2.png) 2. This will download a `.zip` file to your computer. 3. Extract the contents of the `.zip` file. This root folder is the one you’ll refer to from VSCode. It contains a `pom.xml` file and a `src` folder. 4. Open VSCode and select **File > Open Folder** ![VS Code File menu with Open Folder highlighted](../../assets/blog/how-to-move-your-code-from-the-dataweave-playground-to-visual-studio-code-3.png) 5. Select the root folder from step 3. 6. You should now have a structure like the following one. ![VS Code Explorer tree of the exported project showing main.dwl, payload.json, and pom.xml](../../assets/blog/how-to-move-your-code-from-the-dataweave-playground-to-visual-studio-code-4.png) ## Configure your layout 1. You can open both files (`payload.json` and `main.dwl`) and drag-and-drop them to where you want them to be. This will give you a similar layout as the Playground. ![VS Code with payload.json and main.dwl arranged side by side like the Playground layout](../../assets/blog/how-to-move-your-code-from-the-dataweave-playground-to-visual-studio-code-5.png) 2. You can click on the **Explorer** button on the far left if you want to minimize that section. ![Same VS Code layout with the Explorer panel collapsed for more editing space](../../assets/blog/how-to-move-your-code-from-the-dataweave-playground-to-visual-studio-code-6.png) 3. Click on the `main.dwl` window and click on the **Run Preview** button to see the output. ![main.dwl editor with the DataWeave Run Preview button highlighted in the toolbar](../../assets/blog/how-to-move-your-code-from-the-dataweave-playground-to-visual-studio-code-7.png) 4. Success! ![VS Code showing payload, script, and a Preview Output panel rendering "Hello world!"](../../assets/blog/how-to-move-your-code-from-the-dataweave-playground-to-visual-studio-code-8.png) > [!NOTE] > If you don’t get the correct output, make sure you installed Java 8 and the appropriate version of Maven, as well as the latest version of VSCode. ## Enable AutoPreview If you don’t change the AutoPreview settings, you’ll have to click on the **Run Preview** button every time you make a change to the script or the payload. This is useful if you want to save resources and only run the code when you need to update the preview. If you want to enable AutoPreview so it updates automatically when you make changes to the script (as it currently does in the DataWeave Playground), simply: - Open the `main.dwl` file - Right-click on the window - Select **DataWeave: Enable AutoPreview** ![Right-click menu in main.dwl with DataWeave: Enable AutoPreview selected](../../assets/blog/how-to-move-your-code-from-the-dataweave-playground-to-visual-studio-code-9.png) That’s it! You can disable it again if you want using the **Disable** button :) ## More resources As mentioned before, there are a lot more advantages to using the VSCode extension that we didn’t list here. Check out the following resources for more information: - [DataWeave landing page](https://dataweave.mulesoft.com/) - [Developer tutorial] [Getting started with the DataWeave extension for Visual Studio Code](https://developer.mulesoft.com/tutorials-and-howtos/dataweave/dataweave-extension-vscode-getting-started/) - [Developer tutorial] [Getting started with DataWeave libraries in Anypoint Exchange](https://developer.mulesoft.com/tutorials-and-howtos/dataweave/dataweave-libraries-in-exchange-getting-started/) - [Short video tutorial] [How to export a script from the DataWeave Playground to Visual Studio Code](https://youtu.be/_8kCFBdgX5A) - [Twitch live stream] [Exploring products: DataWeave extension for VSCode](https://www.twitch.tv/videos/1525009818) --- ## MuleSoft's Anypoint Flex Gateway logs integration with Grafana Loki Source: https://prostdev.com/post/mulesoft-s-anypoint-flex-gateway-logs-integration-with-grafana-loki | Published: Aug 30, 2022 | Category: Tutorials I’ve been writing about the MuleSoft Flex gateway for the past months, and you can find it here: [https://medium.com/@rcarrascosps](https://medium.com/@rcarrascosps) But one of the most common things to take care of is about consolidating logs and making them part of a greater initiative where you need to have a single pane of glass where you can get information about the health of your system, but also about your services, APIs, microservices, etc. The Flex gateway is going to be part of a greater solution, it will be very improbable that it will be seating just by itself. If you are using it is because you are managing multiple APIs in multiple places. And it is also very probable that your transactions are a mix of APIs calls that goes from one API to another which makes it complicated to trace, diagnose, to monitor. MuleSoft Flex gateway is a very lightweight piece of software that you can configure in both connected mode and local mode. Where the connected mode has all the advantages of managing your APIs from Anypoint Platform, which makes it very appealing. ![Anypoint API Manager dashboard with Total Requests, Policy Violations, Errors and Response Time charts](../../assets/blog/mulesoft-s-anypoint-flex-gateway-logs-integration-with-grafana-loki-2.png) And the local mode is where you do not manage your APIs from Anypoint Platform, but directly in the command line of the system (Linux, Kubernetes, docket) where you’ve installed the gateway. To understand more about those modes you can take a look at my articles or the official documentation: [https://docs.mulesoft.com/gateway/1.1/flex-gateway-overview](https://docs.mulesoft.com/gateway/1.1/flex-gateway-overview) This series of articles (this is the first one) is targeted to integrate the Flex Gateway with Grafana Labs. In particular with Loki and Tempo. And this first article is related to Loki. Grafana Loki is part of the different components that Grafana Labs offer for log consolidation. It can be integrated with a lot of popular open-source and enterprise solutions such as Elasticsearch (logstash), Splunk, etc. You can learn from Grafana Loki here: [https://grafana.com/oss/loki/](https://grafana.com/oss/loki/) What we will learn in this short article is how to send the Flex Gateway logs (at least the agent log) into Grafana Loki and then analyze them, search them, and explore them from the Grafana console. ## Prerequisites The prerequisites for following this article are - Anypoint Platform instance - Install a Flex Gateway either in connected or local mode - Have access to the Flex Gateway logs - Grafana Cloud free-tier account - Promptail agent to point it to Flex Gateway - And that’s it Let’s go step by step and first install a Flex Gateway into a Linux System. ## Install a Flex Gateway instance Go to [https://anypoint.mulesoft.com](https://anypoint.mulesoft.com/), head into Runtime Manager, and click on Flex Gateways. ![Runtime Manager left menu with the Flex Gateways item highlighted](../../assets/blog/mulesoft-s-anypoint-flex-gateway-logs-integration-with-grafana-loki-3.png) Then add a gateway and choose Linux (I will use that option for now). ![Add a Flex Gateway OS picker with Linux, Docker and Kubernetes options, Linux selected](../../assets/blog/mulesoft-s-anypoint-flex-gateway-logs-integration-with-grafana-loki-4.png) And then follow the instructions listed there, which are the following. > [!NOTE] > these commands apply to Flex Gateway version 1.1.0. If you want to work on a different version, the instructions might be different. ```bash curl -XGET https://flex-packages.anypoint.mulesoft.com/ubuntu/pubkey.gpg | sudo apt-key add - echo "deb https://flex-packages.anypoint.mulesoft.com/ubuntu $(lsb_release -cs) main" \ | sudo tee /etc/apt/sources.list.d/mulesoft.list sudo apt update sudo apt install -y flex-gateway ``` Then execute this and change <gateway-name> with any name you choose. The token and organization IDs are going to be different than the ones listed here because they have your own credentials: ```bash sudo flexctl register \ --token=ffb21bee-a718-77d56750e88 \ --organization=f7-4c08-96a3-c9013a01d871 \ --connected=true \ --output-directory=/usr/local/share/mulesoft/flex-gateway/conf.d ``` And finally, start it and check its status: ```bash sudo systemctl start flex-gateway systemctl list-units flex-gateway* ``` Pretty much that should take less than 4 minutes. Just that simple. ## Get access to the Flex Gateway logs Now that the gateway is installed, you can check from the Linux system the log files of it. Let’s focus on one specific which is the gateway log file. And in the next articles, I will focus on the specific APIs logs. Every API can be configured to have its log file, and we will see it in our next article. But to keep things simple we will use the Gateway log. To get access to it, create a directory. For example: ```bash mkdir $HOME/flex-gateway/logs ``` Inside this directory, execute: ```bash journalctl -u flex-gateway-* > flexlog.log ``` You will have the log file in $HOME/flex-gateway/logs/flexlog.out And you will get something like this: ![Flex Gateway agent log lines captured from journalctl into the flexlog.log file](../../assets/blog/mulesoft-s-anypoint-flex-gateway-logs-integration-with-grafana-loki-5.png) Now let’s configure Grafana Loki. ## Grafana Loki configuration You need a Grafana free tier account. You can get it here: [https://grafana.com/products/cloud/](https://grafana.com/products/cloud/) Once you get your account, go to the cloud console (for example [https://grafana.com/orgs/](https://grafana.com/orgs/)<YourOrganizationName>) And there, click on this: ![Grafana Cloud Stack console with Grafana Launch and a highlighted Loki Send Logs button](../../assets/blog/mulesoft-s-anypoint-flex-gateway-logs-integration-with-grafana-loki-6.png) Click on the Send Logs blue button and follow these instructions: ![Grafana Promtail setup instructions showing the config.yaml and the docker run command](../../assets/blog/mulesoft-s-anypoint-flex-gateway-logs-integration-with-grafana-loki-7.png) What we’ve just done is configure the promtail agent to send logs into Grafana Loki. That is a very lightweight agent that in this case is running inside docker. The configuration is very simple, we just need to identify the directory of the log files that we want to send to Loki, and that’s it: ```bash docker run --name promtail --volume "$PWD/promtail:/etc/promtail" --volume "$HOME/flex-gateway/logs:/var/log" grafana/promtail:master -config.file=/etc/promtail/config.yaml ``` The output for that is: ![Promtail container startup logs showing it tailing /var/log/flexlog.log](../../assets/blog/mulesoft-s-anypoint-flex-gateway-logs-integration-with-grafana-loki-8.png) And if we go to Grafana Explorer, and select the Loki logs data source: ![Grafana Explore query builder with the Loki logs data source and a filename label filter](../../assets/blog/mulesoft-s-anypoint-flex-gateway-logs-integration-with-grafana-loki-9.png) Write that simple query in the builder mode or the coding mode: ![LogQL query in the Log browser filtering on filename equals /var/log/flexlog.log](../../assets/blog/mulesoft-s-anypoint-flex-gateway-logs-integration-with-grafana-loki-10.png) And you will get this: ![Grafana Explore showing the Flex Gateway log lines and volume graph returned by the query](../../assets/blog/mulesoft-s-anypoint-flex-gateway-logs-integration-with-grafana-loki-11.png) And now you can start to imagine how much you can do, once you have it in Grafana. As I’ve mentioned, this is a very simple example. The true value will be when we start reading the APIs’ specific log files. That will be in the second article of this series. --- ## Spring Module Integration in a Mule Application (Mule 4) Source: https://prostdev.com/post/spring-module-integration-in-a-mule-application-mule-4 | Published: Jul 26, 2022 | Category: Tutorials The Spring module enables Mule apps to use the Spring framework. In this article, we will use a database datasource that is created by spring beans. Also, we will invoke one bean’s method from a Mule flow. ## Step by step 1 - **Create Mule Project**: Go to Anypoint Studio and create a new Mule project. 2 - **Add spring module**: In Anypoint Studio, the Spring module is provided in the default configuration. In the Mule Palette, click on add modules, search for Spring and add this module to your project. ![Mule Palette Add Modules view with Spring listed among the available modules](../../assets/blog/spring-module-integration-in-a-mule-application-mule-4-2.png) 3 - **Spring config**: Go to spring config, add name as `Spring_Config` and provide files as `beans.xml` ![Spring Config named Spring_Config with beans.xml set as its configuration file](../../assets/blog/spring-module-integration-in-a-mule-application-mule-4-3.png) 4 - **Spring Beans**: Create a `beans.xml` file under `src/main/resources`. Add the below configuration in this file. ```xml ``` 5 - **Spring JDBC**: Add spring JDBC and PostgreSQL dependency and shared library in `pom.xml` ```xml org.springframework spring-jdbc 5.3.2 org.postgresql postgresql 42.3.4 ``` Add below shared library in the shared libraries tag. ```xml org.springframework spring-jdbc org.postgresql postgresql ``` 6 - **Spring JDBC Beans**: Add spring Datasource bean configuration in `beans.xml`. ```xml ``` 7 - **Application Properties**: Add below application properties for JDBC connection in `src/main/resources`. You can replace the correct username, password, and database name in the below properties. ```properties spring.datasource.url=jdbc:postgresql://localhost:5432/spring spring.datasource.username=username spring.datasource.password=password ``` 8 - **Database Config**: Go to Mule Palette and drag the database connector. Configure the database configuration in `global.xml` with the datasource and select the PostgreSQL jar for the JDBC driver. ![Database Config using a Data Source Reference Connection pointing to the dataSource bean](../../assets/blog/spring-module-integration-in-a-mule-application-mule-4-4.png) Test the configuration, this should return a successful connection. 9 - **HTTP Listener for accounts**: Add an HTTP listener and configure it with the default settings. Add a Select DB configuration and a Select query for the accounts table. Make sure the account table is created in your database and there will be at least one entry for the account table. Add a transformer to give the result in JSON format. ![Mule flow with Listener, a database Select, and Transform Message returning JSON](../../assets/blog/spring-module-integration-in-a-mule-application-mule-4-5.png) 10 - **Invoke**: Run the mule application and invoke the endpoint to see the result. This will return records from the accounts table. ![Postman GET to /accounts returning 200 OK with a JSON account record from the database](../../assets/blog/spring-module-integration-in-a-mule-application-mule-4-6.png) 11 - **Create Spring Beans**: Create User POJO class, UserService interface, and UserServiceImpl service implementation class. ```java title="User.java" package com.shyam.model; public class User { private String firstName; private String lastName; private String email; public String getFirstName() { return firstName; } public void setFirstName(String firstName) { this.firstName = firstName; } public String getLastName() { return lastName; } public void setLastName(String lastName) { this.lastName = lastName; } public String getEmail() { return email; } public void setEmail(String email) { this.email = email; } } ``` ```java title="UserService.java" package com.shyam.service; public interface UserService { public String sayHello(); } ``` ```java title="UserServiceImpl.java" package com.shyam.service; import com.shyam.model.User; public class UserServiceImpl implements UserService { private User user; @Override public String sayHello() { return "Hello from " + user.getFirstName(); } public User getUser() { return user; } public void setUser(User user) { this.user = user; } } ``` 12 - **Spring Bean Configuration**: Add below spring bean configuration for User and UserServiceImpl class in `beans.xml`. ```xml ``` 13 - **Mule flow**: From a Mule flow it's very simple to access one of the previously created beans. It just uses an Invoke component to call the Spring bean’s function. ![Mule flow with Listener, a Java Invoke component, and a Logger](../../assets/blog/spring-module-integration-in-a-mule-application-mule-4-7.png) 14 - **JAVA Invoke**: In the Invoke component in the Mule application, we are simply calling the sayHello() method of the UserServiceImpl. Here is the screenshot of the configuration of the Invoke component: ![Invoke component config calling sayHello() on the userService instance of UserService](../../assets/blog/spring-module-integration-in-a-mule-application-mule-4-8.png) 15 - **Invoke**: Invoke the endpoint and see this will return Hello from Shyam, which we have configured the first name in beans property. ![Postman GET to /spring returning 200 OK with the body "Hello from Shyam"](../../assets/blog/spring-module-integration-in-a-mule-application-mule-4-9.png) ## GitHub repository - [https://github.com/shyamrajprasad/mule-spring-integration](https://github.com/shyamrajprasad/mule-spring-integration) ## References - [https://docs.mulesoft.com/spring-module/1.3/](https://docs.mulesoft.com/spring-module/1.3/) - [https://dzone.com/articles/using-spring-beans-in-mule](https://dzone.com/articles/using-spring-beans-in-mule) - [https://shyamrajprasad.medium.com/spring-module-integration-in-mule-application-8de5a08303e6](https://shyamrajprasad.medium.com/spring-module-integration-in-mule-application-8de5a08303e6) --- ## JWT token creation using DataWeave Source: https://prostdev.com/post/jwt-token-creation-using-dataweave | Published: Jul 19, 2022 | Category: Tutorials Security implementations have been revolutionary through OAuth 2.0, OpenID Connect, SAML, etc. OAuth 2.0 and OpenID connect mostly use JWT as a token format. JWT is a very familiar term for the API fraternity. There are instances where we need to create a [JWT token](https://jwt.io/introduction) to authorize our APIs to a service or authorize any client if we are using any custom solutions for authentication/authorization. Let’s take a deeper look at JWT tokens. ## Sample JWT token ``` ewogICJhbGciOiAiSFMyNTYiLAogICJ0eXAiOiAiSldUIiwKICAia2lkIjogIjEwMSIKfQ.ewogICJpc3MiOiAiR1RBIiwKICAiaWF0IjogMTY1NjQyMTQ0NiwKICAicmVxdWVzdGVkU2NvcGUiOiBbCiAgICAieG90cCIKICBdCn0.l2slJ86T7J3at9UG5esKMi5B9h02WjcpIuMZm_5mxzM ``` Let's see the basic structure of this token. (Visit [https://jwt.io/#debugger-io](https://jwt.io/#debugger-io)) ![jwt.io debugger decoding the token into its header, payload and signature parts](../../assets/blog/jwt-token-creation-using-dataweave-2.png) As you can see in the image, the token is decoded into three parts, 1. **Header** ``` ewogICJhbGciOiAiSFMyNTYiLAogICJ0eXAiOiAiSldUIiwKICAia2lkIjogIjEwMSIKfQ ``` 2. **Payload** ``` ewogICJpc3MiOiAiR1RBIiwKICAiaWF0IjogMTY1NjQyMTQ0NiwKICAicmVxdWVzdGVkU2NvcGUiOiBbCiAgICAieG90cCIKICBdCn0 ``` 3. **Signature** ``` l2slJ86T7J3at9UG5esKMi5B9h02WjcpIuMZm_5mxzM ``` We will see in the following topic how to create this token. ## How to create JWT using Java There are multiple libraries to do this in JAVA: I always try to use the simplest code. ```java import java.nio.charset.StandardCharsets; import java.security.InvalidKeyException; import java.security.NoSuchAlgorithmException; import java.util.Base64; import javax.crypto.Mac; import javax.crypto.spec.SecretKeySpec; import org.json.JSONException; import org.json.JSONObject; public class JWT { public static String encodeString(byte[] bytes) { return Base64.getUrlEncoder().withoutPadding().encodeToString(bytes); } public static String encodeJSON(JSONObject obj) { return encodeString(obj.toString().getBytes(StandardCharsets.UTF_8)); } public static String createJWT(String data, String secret) throws NoSuchAlgorithmException { try { byte[] hash = secret.getBytes(StandardCharsets.UTF_8); Mac sha256Hmac = Mac.getInstance("HmacSHA256"); SecretKeySpec secretKey = new SecretKeySpec(hash, "HmacSHA256"); sha256Hmac.init(secretKey); byte[] signedBytes = sha256Hmac.doFinal(data.getBytes(StandardCharsets.UTF_8)); return encodeString(signedBytes); } catch (InvalidKeyException ex) { System.out.print("FAILED to sign"); return null; } } public static String JWTtoken(String[] args) throws JSONException, NoSuchAlgorithmException { String jwtHeader = "{\n \"alg\": \"HS256\",\n \"typ\": \"JWT\",\n \"kid\": \"101\"\n}"; String jwtPayload = "{\n \"iss\": \"GTA\",\n \"iat\": 1656422976,\n \"requestedScope\": [\n \"xotp\"\n ]\n}"; String secret = "1A3B4A5CE86E0BF3AF6FF575C93438BH"; String signedData = createJWT( encodeJSON(new JSONObject(jwtHeader)) + "." + encodeJSON(new JSONObject(jwtPayload)), secret); String jwtToken = encodeJSON(new JSONObject(jwtHeader)) + "." + encodeJSON(new JSONObject(jwtPayload)) + '.' + signedData; return jwtToken; } public static void main(String args[]) { try { JWTtoken(null); } catch (NoSuchAlgorithmException e) { e.printStackTrace(); } catch (JSONException e) { e.printStackTrace(); } } } ``` The token generated: ``` eyJraWQiOiIxMDEiLCJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJHVEEiLCJyZXF1ZXN0ZWRTY29wZSI6WyJ4b3RwIl0sImlhdCI6MTY1NjQyMjk3Nn0.3kM0_DUuJCIig2OlyqxqnjODUxRvTA93d9Tgu-ZhOgw ``` Let’s decode this token. ![jwt.io decoding the Java-generated token, showing the GTA issuer and xotp scope payload](../../assets/blog/jwt-token-creation-using-dataweave-3.png) ## How to create JWT using DataWeave JAVA seems to be very technical. Let’s switch to our magical language. ```dataweave %dw 2.0 //Imports import * from dw::Crypto import * from dw::core::Binaries import * from dw::core::URL //Variables var secret = "1A3B4A5CE86E0BF3AF6FF575C93438BH" //256 bit key var tokenHeader = { "alg": "HS256", // The algorithm to sign "typ": "JWT", // Type of token "kid": "101" // Key Id } var tokenPayload = { "iss": "GTA", // Issuer of JWT "iat": 1656422976, // now() as Number {unit: "seconds"}, // Time in seconds since epoch, hardcoded to match with JAVA code block "requestedScope": ["xotp"] // Request scope } // Function to encode data into base64 // Note:Web Tokens use Base64Url instead of the typical Base64. They are basically the same except Base64Url are safe to pass in a URL because they use – instead of + and _ instead of / and they omit the = padding characters at the end of the string. You can do string replacement to properly convert the token then call fromBase64. // To read more on encoding/decoding base64 formats visit - https://docs.mulesoft.com/dataweave/2.4/dw-binaries-functions-tobase64 fun base64encodeURL(data) = toBase64(data) replace "/" with ("_") replace "+" with ("-") replace "=" with "" //Variables var header = base64encodeURL(write(tokenHeader, "application/json")) // Convert to Stringified JSON var payload = base64encodeURL(write(tokenPayload, "application/json")) // Convert to Stringified JSON var signature = base64encodeURL(HMACBinary(secret as Binary, (base64encodeURL(header) ++ "." ++ base64encodeURL(payload)) as Binary, "HmacSHA256")) output application/json --- header ++ "." ++ payload ++ "." ++ signature ``` The token generated: ``` ewogICJhbGciOiAiSFMyNTYiLAogICJ0eXAiOiAiSldUIiwKICAia2lkIjogIjEwMSIKfQ.ewogICJpc3MiOiAiR1RBIiwKICAiaWF0IjogMTY1NjQyMjk3NiwKICAicmVxdWVzdGVkU2NvcGUiOiBbCiAgICAieG90cCIKICBdCn0._RUefLwi3UBP7jbxE7VHB-t-aMmCNdvFSr7frgW7wNY ``` Let’s decode this token: ![jwt.io decoding the DataWeave-generated token to the same header and payload data](../../assets/blog/jwt-token-creation-using-dataweave-4.png) Voila, you are done. TBH, it's way more fun to code in DataWeave. The tokens generated from the JAVA class and DataWeave get decoded to the same data. ## Summary JWT is vital in today’s API world. It's not enough to just know the code; we need to focus on the security of API from all perspectives. ## References - [https://jwt.io/introduction](https://jwt.io/introduction) - [https://docs.mulesoft.com/dataweave/2.4/dw-crypto](https://docs.mulesoft.com/dataweave/2.4/dw-crypto) --- ## How to Integrate AWS Lambda with MuleSoft Source: https://prostdev.com/post/how-to-integrate-aws-lambda-with-mulesoft | Published: Jul 12, 2022 | Category: Tutorials In this blog post, I will demonstrate how to Integrate AWS Lambda with MuleSoft. The GitHub repository with the Mule Project can be found at the end of the post. AWS Lambda is an on-demand cloud computing resource offered as a function-as-a-service by AWS. AWS Lambda allows you to add custom logic to AWS resources such as Amazon S3 buckets and Amazon DynamoDB tables, so you can easily apply it to compute data as it enters or moves through the cloud. AWS Lambda was designed for use cases such as image or object uploads to [Amazon S3](https://en.wikipedia.org/wiki/Amazon_S3), updates to [DynamoDB](https://en.wikipedia.org/wiki/DynamoDB) tables, responding to website clicks, or reacting to sensor readings from an [IoT](https://en.wikipedia.org/wiki/Internet_of_Things)-connected device. AWS Lambda can also be used to automatically provision back-end services triggered by custom [HTTP requests](https://en.wikipedia.org/wiki/HTTP_request), and "spin down" such services when not in use, to save resources. Anypoint Connector for Amazon Lambda (Amazon Lambda Connector) enables you to execute AWS Lambda operations within your Mule flows. You can then use the response of the Lambda operation to process another Mule flow as needed. In this article, we will see how we can use the Anypoint Lambda Connector to Integrate. For more information on Amazon Lambda Connector 1.0 - Mule 4, refer to this [link](https://docs.mulesoft.com/amazon-lambda-connector/1.0/). ## 1. Create an AWS Free tier account Open the link [https://aws.amazon.com/console/](https://aws.amazon.com/console/). The below page opens up: ![AWS Management Console landing page with a Sign In to the Console button](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-2.png) Click **Sign In to the Console** and then create a new AWS account. ![AWS sign-in page with the Create a new AWS account button highlighted](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-3.png) Provide the required details and click on **Verify email address.** ![Sign up for AWS form asking for root user email address and AWS account name](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-4.png) A verification code will be sent to your email. Enter the code and click on **Verify**. ![AWS "Verify your email address" email containing a verification code](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-5.png) ![AWS sign-up "Confirm you are you" screen with a field to enter the verification code](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-6.png) Once the verification is successful, provide the password and click on **continue (step 1 of 5)**. ![AWS sign-up Create your password step after the email was successfully verified](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-7.png) Provide all the contact information and click on **continue (step 2 of 5)**. ![AWS sign-up Contact Information form for full name, phone number, and address](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-8.png) It asks for credit card information. You can safely provide as you won't be charged unless you exceed the free tier limit. For more information on this please watch this video on YouTube ([AWS Free Tier Overview](https://www.youtube.com/watch?v=Qou3oKcvyww&t=3s)). After this step click on **Create Account**. After this sign in to the AWS console by selecting the Root user. ![AWS sign-in page with the Root user option selected](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-9.png) You will now be logged in to the AWS Console as shown below: ![AWS Console Home showing recently visited services including IAM, Lambda, and S3](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-10.png) ## 2. Manage Access to AWS resources First, select the region of your choice from the list given below: ![AWS region dropdown open, listing US and Asia Pacific regions](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-11.png) Click on **Services** -> **Security, Identity & Compliance** -> **IAM.** ![AWS Services menu open on Security, Identity and Compliance with IAM listed](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-12.png) Click on **Users** and on **Add Users.** ![IAM Users list showing one user with an Add users button](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-13.png) Provide the User Name and Select AWS Credential type as shown below: ![IAM Add user step 1 setting a user name and selecting programmatic access key](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-14.png) Click on **Next: Permissions** and on **Attach existing policies directly.** ![IAM permissions step with Attach existing policies directly and AWSLambda search](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-15.png) Click on **AWSLambda_FullAccess** and **Next: Tags.** ![Policy list with AWSLambda_FullAccess checkbox selected](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-16.png) Click on **Review** and click on **Create User.** ![IAM Review screen summarizing the user with AWSLambda_FullAccess attached](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-17.png) The User is created successfully. Make a note of the Access Key ID and Secret access key as you cannot retrieve them again. ![IAM success screen showing the new user's Access key ID and Secret access key](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-18.png) ![IAM user detail page showing AWSLambda_FullAccess attached directly](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-19.png) ## 3. Create a Lambda Function in AWS In the AWS console search Lambda and click on the search result Lambda. ![AWS console search for Lambda showing the Lambda service result](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-20.png) Click on **Create Function.** ![AWS Lambda Functions page with a Create function button](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-21.png) I used a simple “hello world” blueprint. ![Create function screen with Use a blueprint and hello-world blueprint selected](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-22.png) Provide all the basic information and click on **Create Function**. ![Lambda Basic information form setting a function name and execution role](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-23.png) ![Lambda Functions list showing the created test-hello-world-mule-poc function on Node.js 12.x](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-24.png) ![Lambda function overview page showing the function ARN and Add trigger options](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-25.png) The Source Code of the function can be seen below: ![Lambda Code source editor showing the index.js hello world handler](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-26.png) You can execute the script by clicking on **Test** and the execution result is as shown below: ![Lambda test execution result showing status Succeeded with function logs](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-27.png) ## 4. Configure Amazon Lambda Connector in MuleSoft First, we need to create a new project in Anypoint Studio and manage dependencies as shown below: ![Anypoint Studio project right-click menu opening Manage Dependencies then Manage Modules](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-28.png) Add the **Amazon Lambda Connector** dependency -> **Finish** -> **Apply and close.** ![Add Dependencies dialog with Amazon Lambda Connector 1.0.4 added to selected modules](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-29.png) Now, this dependency will be added to the `pom.xml`. ![pom.xml showing the mule4-amazon-lambda-connector dependency added](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-30.png) You will now be able to see all the Amazon Lambda Connectors available in the Mule Palette. ![Mule Palette search for Lambda listing Get Function, Invoke, Invoke Async and List Functions](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-31.png) ## 5. How to use the Invoke, Get Function, and List Functions Lambda Connectors More information on these connectors can be found at [https://docs.mulesoft.com/amazon-lambda-connector/1.0/amazon-lambda-connector-reference](https://docs.mulesoft.com/amazon-lambda-connector/1.0/amazon-lambda-connector-reference) ### Invoke Connector "Invokes a Lambda function. You can invoke a function synchronously (and wait for the response), or asynchronously. To invoke a function asynchronously, set InvocationType to Event. For synchronous invocation, details about the function response, including errors, are included in the response body and headers. For either invocation type, you can find more information in the execution log and trace." More information at [this link](https://docs.mulesoft.com/amazon-lambda-connector/1.0/amazon-lambda-connector-reference#Create20150331FunctionsInvocationsByFunctionName). ![Mule flow with Invoke operation configured with function name and request-response invocation type](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-32.png) The access key and the secret key are obtained as shown in the “Manage Access to AWS resources” step. ![Amazon Lambda Connector global config with Access Key, Secret Key and us-east-1 region](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-33.png) Now let us test this using Postman. ![Postman calling lambdaInvoke with a JSON body returning value1 and a 200 OK status](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-34.png) ### Get Function Connector "Returns information about the function or function version, with a link to download the deployment package that’s valid for 10 minutes. If you specify a function version, only details that are specific to that version are returned." More information at [this link](https://docs.mulesoft.com/amazon-lambda-connector/1.0/amazon-lambda-connector-reference#Get20150331FunctionsByFunctionName). ![Mule flow with the Get Function operation configured with the function name](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-35.png) The response from the Get Function is as shown below. ![Mule Debugger showing the Get Function JSON response with function configuration details](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-36.png) ![Postman calling lambdaGet returning the function Configuration JSON](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-37.png) ### List Functions Connector "Returns a list of Lambda functions, with the version-specific configuration of each function. Lambda returns up to 50 functions per call. Set FunctionVersion to ALL to include all published versions of each function in addition to the unpublished version. The ListFunctions action returns a subset of the FunctionConfiguration fields." More information at [this link](https://docs.mulesoft.com/amazon-lambda-connector/1.0/amazon-lambda-connector-reference#Get20150331Functions). ![Mule flow with the List Functions operation configured with empty optional fields](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-38.png) ![Postman calling lambdaList returning a Functions array of Lambda functions](../../assets/blog/how-to-integrate-aws-lambda-with-mulesoft-39.png) ## Conclusion This is how we can easily create functions in AWS Lambda and Integrate with MuleSoft using the **Amazon Lambda Connectors**. ## Best practices For the purpose of simplification, I have mentioned the configuration details directly. But it is always recommended to externalize these details and encrypt the sensitive data. ## GitHub repository - [https://github.com/ProstDev/aws-lambda-mulesoft](https://github.com/ProstDev/aws-lambda-mulesoft) Thanks for reading my article and I hope it will be of some use! See you in the next post! -Pravallika --- ## Tracing Module in Mule 4 Source: https://prostdev.com/post/tracing-module-in-mule-4 | Published: Jul 5, 2022 | Category: Tutorials The tracing module enables you to enhance your logs by adding, removing, and clearing all variables from the logging context for a given Mule event. It also enables you to modify the correlation ID during flow execution. ## Uses Of Tracing Module - Clear all the logging variables from the event logging context. - Remove a logging variable from the logging context. - Set logging variables to logging context. - Modify the correlation ID during flow execution ![Mule Palette listing Tracing module operations: Clear, Remove, Set logging variable, and With CorrelationID](../../assets/blog/tracing-module-in-mule-4-2.png) ## Tracing Module Application 1 - **Create Mule application**: Go to Anypoint studio and create a mule project with the name `mulesoft-tracing-module-integration`. 2 - **Add Tracing module**: In the mule palette view, click on **+ Add Modules** and search for tracing modules. Drag the Tracing module to your modules. 3 - **HTTP Listener**: Create an HTTP listener with the default configuration and specify the `/trace` base path. 4 - **MDC Logging**: Mapped Diagnostic Context (MDC) enriches logging and improves tracking by providing more context or information in the logs for the current Mule event. By default, Mule logs two MDC entries: `processor`, which shows the location of the current event; and `event`, which shows the correlation ID of the event. Open the `log4j.xml` file and replace `[processor: %X{processorPath}; event: %X{correlationId}]` with `[%MDC]`. 5 - **Create set variable logging**: Create a set logging variable for `customerId` and request path as below. In the logger, we will print the payload. ![Mule flow with Listener, Set logging variable for customerId and requestPath, then a Logger](../../assets/blog/tracing-module-in-mule-4-3.png) ```xml ``` 6 - **Run application**: Run the application and test it out with the below command. ```bash curl --location --request POST 'http://localhost:8081/trace' \ --header 'Content-Type: application/json' \ --data-raw '{ "orderId": 54810284, "customerId": "CUST-12934", "items": [ "P-123", "P-452" ] }' ``` 7 - **Verify logs**: Verify the logs `customerId` and request path will be printed in the context path logs. The below log will be printed in the file location mentioned in `log4j.xml`. ```text INFO 2022-05-08 21:46:29,898 [[MuleRuntime].uber.07: [mulesoft-tracing-module-integration].mulesoft-tracing-module-integrationFlow.CPU_LITE @69de6b8b] [{correlationId=37e4bf90-ceea-11ec-9c4a-f84d89960c47, customerId=ARG-12934, processorPath=mulesoft-tracing-module-integrationFlow/processors/2, requestPath=POST:/trace}] logging variables: { "orderId": 548102842, "customerId": "ARG-12934", "items": [ "CP-123", "CP-452" ] } ``` > [!NOTE] > You will see different logs in the console with `processor` and `event`. 8 - **Console log4j changes**: Open `log4j.xml`, add Console logging in Appenders tag. Also, add this reference to the AsyncRoot tag. ```xml ``` 9 - **Run application**: Re-run the Mule application and verify the logger. The same logging will be printed in the console. 10 - **Remove logging variable**: After the logger, let’s remove one of the logging variables i.e request path. Also after this, remove the logging variable, and add the logger for payload. ```xml ``` ![Mule flow extended with a Remove logging variable for requestPath followed by another Logger](../../assets/blog/tracing-module-in-mule-4-4.png) 11 - **Run application and verify**: Re-run the application, hit the endpoint, and verify the logs, you will see the request path will be removed from the context path. When you will hit the application multiple times, you will see different correlation IDs. ```text INFO 2022-05-08 22:08:11,518 [[MuleRuntime].uber.09: [mulesoft-tracing-module-integration].mulesoft-tracing-module-integrationFlow.CPU_LITE @11ace7f9] [{correlationId=3fd22371-ceed-11ec-9c4a-f84d89960c47, customerId=ARG-12934, processorPath=mulesoft-tracing-module-integrationFlow/processors/4}] logging without request path: { "orderId": 548102842, "customerId": "ARG-12934", "items": [ "CP-123", "CP-452" ] } ``` 12 - **Clear logging variable**: After the payload logger, let's add a clear logging variable, it will remove all the logging variables which were created during the flow. ![Mule flow with a Clear logging variables component added before the final Logger](../../assets/blog/tracing-module-in-mule-4-5.png) 13 - **Re-run application and verify**: Run the application again and hit the endpoint and verify the logs. There will be no `customerId` logging variable. ```text INFO 2022-05-08 22:08:11,520 [[MuleRuntime].uber.09: [mulesoft-tracing-module-integration].mulesoft-tracing-module-integrationFlow.CPU_LITE @11ace7f9] [{correlationId=3fd22371-ceed-11ec-9c4a-f84d89960c47, processorPath=mulesoft-tracing-module-integrationFlow/processors/6}] no logging variable :: { "orderId": 548102842, "customerId": "ARG-12934", "items": [ "CP-123", "CP-452" ] } ``` 14 - **With Correlation ID**: Let’s add with correlation id after payload logger, and change correlation Id. I have changed the correlation id to append my name. Add the payload logger into this too. ![With CorrelationID config setting Correlation id to the expression correlationId ++ "shyam"](../../assets/blog/tracing-module-in-mule-4-6.png) 15 - **Run the application**: Re-run the application, hit the URL again, and verify the logs with the correlation logger. This updated correlation id will be visible only for the logger available under the correlation ID. ```text INFO 2022-05-08 22:20:33,690 [[MuleRuntime].uber.02: [mulesoft-tracing-module-integration].mulesoft-tracing-module-integrationFlow.CPU_LITE @3b8d37c5] [{correlationId=fa14acc0-ceee-11ec-b249-f84d89960c47: shyam, processorPath=mulesoft-tracing-module-integrationFlow/processors/7/processors/0}] with changed correlation id:: { "orderId": 548102842, "customerId": "ARG-12934", "items": [ "CP-123", "CP-452" ] } ``` 16 - **Add payload logger**: Add payload logger at the last, you can re-run the application and the updated correlation id will not be available to this logger. ```text INFO 2022-05-08 22:20:33,696 [[MuleRuntime].uber.04: [mulesoft-tracing-module-integration].mulesoft-tracing-module-integrationFlow.CPU_LITE @3b8d37c5] [{correlationId=fa14acc0-ceee-11ec-b249-f84d89960c47, processorPath=mulesoft-tracing-module-integrationFlow/processors/8}] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: { "orderId": 548102842, "customerId": "ARG-12934", "items": [ "CP-123", "CP-452" ] } ``` ## GitHub repository - [https://github.com/shyamrajprasad/mulesoft-tracing-module-integration](https://github.com/shyamrajprasad/mulesoft-tracing-module-integration) ## References - [https://docs.mulesoft.com/tracing-module/1.0/](https://docs.mulesoft.com/tracing-module/1.0/) --- ## How to Invert a Binary Tree with DataWeave Source: https://prostdev.com/post/how-to-invert-a-binary-tree-with-dataweave | Published: Jun 7, 2022 | Category: Tutorials The [DataWeave language](https://docs.mulesoft.com/mule-runtime/latest/dataweave) is a simple, powerful tool used to query and transform data inside of Mule. But you can also use it to solve algorithmic problems. How to invert a Binary Tree is a popular coding question to test your skills. Let’s try to solve this with DataWeave. ## Definition Ok, but what is a binary tree? Yes, sure! Firstly we define a **tree** as a set of elements (nodes), with the following properties: - Each node can be connected to many children - Each node has one parent, except the root node that has no parent A **binary tree** is a tree where each node has at most 2 children: “left” and “right”. We can define the DataWeave **binary tree** structure in the following way: ```dataweave type Tree= { value: String, right?: Tree, left?: Tree } ``` ## Problem In order to invert a binary tree, we need to swap left and right for every level of the tree, as you can see in the following example: ![Diagram of a 5-node binary tree mirrored so left and right children swap at each level](../../assets/blog/how-to-invert-a-binary-tree-with-dataweave-2.png) ## Solution It's easier than it looks. Thanks to Mr. Recursion! We can solve the problem for the first children by swapping them and then recursively applying the same solution for the left and right sub-tree. **Input** ```json { "value": "2", "left": { "value": "1" }, "right": { "value": "4", "left": { "value": "3" }, "right": { "value": "5" } } } ``` **DataWeave Solution** ```dataweave %dw 2.0 output application/json type Tree= { value: String, left?: Tree, right?: Tree } fun invertTree(node ) : Tree = { value: node.value , ( left: invertTree(node.right )) if (node.right?), ( right: invertTree(node.left )) if (node.left?) } --- invertTree(payload) ``` **Output** ```json { "value": "2", "left": { "value": "4", "left": { "value": "5" }, "right": { "value": "3" } }, "right": { "value": "1" } } ``` > [!NOTE] > This solution is using regular recursive functions. If the recursion level exceeds 255 iterations, you will receive a Stack Overflow error. To avoid this, you can also use tail-recursive functions. To learn more about this, see [this article](https://blogs.mulesoft.com/dev-guides/how-to-tutorials/untie-multilevel-structures-dataweave-recursive-calls/). ## Conclusion Now we know how to use Mr. Recursion in DataWeave in order to solve the invert binary tree problem. Thanks for reading my post and I hope it will help understand this wonderful language. That’s it for now! See you in the next post! -Stefano --- ## Universal API Management, Anypoint Flex Gateway, and API Governance Source: https://prostdev.com/post/universal-api-management-anypoint-flex-gateway-and-api-governance | Published: May 31, 2022 | Category: Guides As you are aware that recently MuleSoft has introduced Universal API Management capabilities via Flex Gateway and API Governance components as part of Anypoint Platform. These two components are very important as a part of API lifecycle management and those will help to manage any APIs using a single platform and ensure that whatever API specs that you are designing are with best practices and all security aspects have been taken care of during API design. ## Universal API Management Universal API Management allows you to manage, govern, or secure the APIs within a single control plane, it doesn’t matter whether they are Mule or non-Mule APIs or where they are located (on-premise, cloud, or anywhere). - It will allow the organizations or enterprises to control, manage, and secure the APIs under a single umbrella. - Adapt any architecture with a lightweight and flexible API Gateway to manage and secure the APIs. - Govern all APIs under a single platform. ## Anypoint Flex Gateway Flex Gateway is ultrafast and manages the APIs running anywhere. - Secure and Manage APIs located anywhere. - Extend Anypoint Platform capabilities to Mule as well as non-Mule APIs. - Achieve consistent security and governance across every API operating in any environment. - Flex Gateway can be used in local or connected mode. **What are the benefits of Flex Gateway?** - Flex gateway is an ultra-fast gateway that can be used for any APIs (MuleSoft or Non-MuleSoft APIs), deployed anywhere (Cloud, Docker, Kubernetes, Customer Hosted, etc.). - Easily manage all the APIs within your organization from a single platform and under a single umbrella. - Extend the Anypoint Platform Capabilities to Mule and non-Mule APIs. - Secure, discover, govern, or engage the APIs (Mule and non-Mule APIs). - Set up the Flex Gateway easily in 2 modes (Local and Connected). - Adapt any architecture with a lightweight and flexible API Gateway to manage and secure the APIs. **Can we apply any out-of-the-box, as well as custom API policies to APIs, published to Flex Gateway?** Yes, you can apply any out-of-the-box as well as custom policies to APIs published to Flex Gateway. You can apply API Manager alerts and view APIs metrics. **Is Flex Gateway part of a 30-day free trial Anypoint Platform account?** Yes, you can find Flex Gateway as part of a 30-day free trial Anypoint Platform account. **Where to find MuleSoft documentation for Flex Gateway overview?** Here is the link: [https://docs.mulesoft.com/gateway/flex-gateway-overview](https://docs.mulesoft.com/gateway/flex-gateway-overview) **How to upgrade Flex Gateway?** Here is the link explaining how to upgrade Flex Gateway: [https://docs.mulesoft.com/gateway/flex-gateway-upgrade](https://docs.mulesoft.com/gateway/flex-gateway-upgrade) **How to uninstall Flex Gateway?** Here is the link explaining how to uninstall Flex Gateway: [https://docs.mulesoft.com/gateway/flex-gateway-uninstall](https://docs.mulesoft.com/gateway/flex-gateway-uninstall) **What is the shared responsibility for Flex Gateway between MuleSoft and you?** Here is the MuleSoft document explaining the shared responsibility model for Flex Gateway: [https://docs.mulesoft.com/gateway/flex-shared-responsibility](https://docs.mulesoft.com/gateway/flex-shared-responsibility) **What authentication mechanism is supported for installing Flex Gateway?** There are three types of authentication mechanisms supported for installing Flex Gateway: - Anypoint Username and Password - Auth Token - Connected App **Where can Flex Gateway be set up?** Flex Gateway can be set up on three different operating systems: - Install Flex Gateway as a Linux Service. - Install Flex Gateway as a Docker Container. - Install Flex Gateway as a Kubernetes Ingress Controller. **What are the different steps for setting up Flex Gateway?** Here is the MuleSoft documentation, showing what commands can be used to set up the Flex Gateway. - [Review Prerequisites](https://docs.mulesoft.com/gateway/flex-review-prerequisites) - [Install Flex Gateway](https://docs.mulesoft.com/gateway/flex-install) - [Run Flex Gateway (Connected Mode)](https://docs.mulesoft.com/gateway/flex-conn-reg-run) - [Run Flex Gateway (Local Mode)](https://docs.mulesoft.com/gateway/flex-local-reg-run) - [Add Replicas (Connected Mode)](https://docs.mulesoft.com/gateway/flex-conn-rep-run) - [Add Replicas (Local Mode)](https://docs.mulesoft.com/gateway/flex-local-rep-run) - [Manage APIs (Connected Mode)](https://docs.mulesoft.com/gateway/flex-conn-manage) - [Manage APIs (Local Mode)](https://docs.mulesoft.com/gateway/flex-local-manage) **Can you use a single Flex Gateway for multiple APIs agnostic of the technology and location where it is deployed?** Yes, we can use a single Flex Gateway for multiple APIs, it doesn’t matter where they are running and in which technologies they have been implemented. Here is a sample architecture showing a single Flex Gateway for multiple APIs. In the above Flex Gateway architecture, we are using a single Flex Gateway in Docker container with multiple replicas to connect multiple APIs implemented in any technology. Flex Gateway replicas actually run in the Docker container in the above architecture and it is registered in Anypoint Platform. **Can you use multiple Flex Gateways for multiple APIs?** Yes, we can use multiple Flex Gateways for multiple APIs deployed anywhere. Here is a sample architecture showing a multiple Flex Gateway for multiple APIs. In the above Flex Gateway architecture, we are using multiple Flex Gateways in Docker containers with multiple replicas to connect multiple APIs implemented in any technology. We are using one Flex Gateway for each API in the above architecture and this can be grouped according to your need and requirements. Flex Gateway replicas actually run in the Docker container in the above architecture and it is registered in Anypoint Platform. Here is the list of videos that will explain how to set up the Flex Gateway in both Connected and Local mode: - [Anypoint Universal API Management and Flex Gateway - Part I | MuleSoft](https://youtu.be/ZG9Si5w-hUA) - [Anypoint Universal API Management and Flex Gateway - Part II | Flex Gateway in Connected Mode](https://youtu.be/t_oevGDU3Ro) - [Anypoint Universal API Management and Flex Gateway - Part III | Manage, Secure API in Connected Mode](https://youtu.be/2dPfR7riEQI) - [Anypoint Universal API Management and Flex Gateway - Part IV | Flex Gateway in Local Mode](https://youtu.be/Ouzt2zyZNkM) - [Anypoint Universal API Management and Flex Gateway - Part V | Manage, Secure API in Local Mode](https://youtu.be/OY2EVsS3SPI) - [Anypoint Universal API Management and Flex Gateway - Part VI | Manage Multiple API in Local Mode](https://youtu.be/2meUTtANR_o) - [Surat MuleSoft Meetup#41 (Flex Gateway and API Governance)](https://youtu.be/ocaeFlBD9dA) ## API Governance MuleSoft has recently introduced API Governance as a part of Anypoint Platform that enables you to apply governance ruleset to your APIs that ensures API Consistency and provides default several rulesets such as a Top 10 OWASP API Security, Anypoint API Best Practices, OpenAPI Best Practices governance rulesets, etc. API Governance will ensure the API designs across the enterprises are consistent and are designed with API best practices and guidelines. This will ensure the security of the API and improve the quality of the APIs. **What are the benefits of API Governance?** - Enable developers to apply governance rulesets at design time. - Produce consistent API specs across the enterprises. - Improved API Quality and Security. - API design with Anypoint best practices and OpenAPI best practices. - Ensure Design-Time conformance. - Reduce Top 10 OWASP security risks. **Are there any default rulesets for API Governance?** API Governance comes with the following default rulesets: - [Anypoint Best Practices](https://anypoint.mulesoft.com/exchange/68ef9520-24e9-4cf2-b2f5-620025690913/anypoint-best-practices/minor/1.0/) - [Authentication Security Best Practices](https://anypoint.mulesoft.com/exchange/68ef9520-24e9-4cf2-b2f5-620025690913/authentication-security-best-practices/minor/1.0/) - [HTTPS Enforcement](https://anypoint.mulesoft.com/exchange/68ef9520-24e9-4cf2-b2f5-620025690913/https-enforcement/minor/1.0/) - [OpenAPI Best Practices](https://anypoint.mulesoft.com/exchange/68ef9520-24e9-4cf2-b2f5-620025690913/openapi-best-practices/minor/1.0/) - [OWASP API Security Top 10 2019 Checklist](https://anypoint.mulesoft.com/exchange/68ef9520-24e9-4cf2-b2f5-620025690913/owasp-api-security/minor/1.0/) - [Required Examples](https://anypoint.mulesoft.com/exchange/68ef9520-24e9-4cf2-b2f5-620025690913/required-examples/minor/1.0/) **How to implement API Governance for the APIs?** The first step for creating API Governance is to create the profile in Anypoint Platform’s API Governance and select what are the rulesets that you need to enable for that profile and also you can add filters and notifications. Filters will ensure which APIs need to be scanned against the profile that we have created. Notifications will generate emails to the users in case the APIs haven't been designed according to rulesets associated with the profile and they will be marked as **Non-Conformant**. There are three statuses maintained for your APIs as part of the API Governance: - **Not Validated** - API is not validated against the API Governance profile. - **Conformant** - API has satisfied the rulesets that were associated with the profile. - **Non-Conformant** - API has not satisfied the rulesets that were associated with the profile. Below is the report generated for your APIs and it will show the status of the APIs with what are the rulesets failed and what are the violations in your APIs. Here is a video showing how to implement Anypoint API Governance: [Anypoint API Governance With MuleSoft](https://youtu.be/BZXbRWJ5Tro). ## Conclusion As we have seen in the above article, how Flex Gateway can be used to manage any APIs agnostic of technologies and platforms. API Governance will ensure that the design of your APIs is consistent, secure, and defined with Anypoint API and OpenAPI best practices and Top 10 OWASP security has been taken care of. These are two important features that have been released recently as a part of Anypoint Platform. Next, you can explore the Flex Gateway and API Governance capabilities within Anypoint Platform. --- ## 3 scenarios to consume an Apache Kafka Topic with MuleSoft Source: https://prostdev.com/post/3-scenarios-to-consume-an-apache-kafka-topic-with-mulesoft | Published: May 10, 2022 | Category: Guides Apache Kafka is one of the best options in the market for data streaming. Developed by LinkedIn engineers and then opened up for the open-source community, it has been very well received (and used) by many organizations. If you need to move/produce streams of data from one point to another, Apache Kafka is the right option for you. Apache Kafka has simple architecture in terms of how the information is streamed: - Topics - Partitions - Replicas - Brokers Those four concepts are the basis for understanding where and how a producer and/or a consumer can produce/read information from Kafka. ![Kafka architecture diagram: producers feeding brokers with topics and partitions, read by consumer groups, with zookeeper](../../assets/blog/3-scenarios-to-consume-an-apache-kafka-topic-with-mulesoft-2.png) The architecture itself, as we’ve mentioned, is not complex, but the zookeeper dependency may be one of the challenges when deploying this on your own. That is why [Confluent](https://www.confluent.io/) is a good option for customers who want to avoid the configuration and maintenance challenges that Apache Kafka may bring to the table. Producers are any type of application and/or platform that needs to produce information into Kafka. For example: - Log files of an application that you would like to send to Kafka for further analysis and/or consolidation - Geolocation of your fleet of buses that needs to be processed and analyzed in real-time - Information from your Point-of-sale (PoS) that you would like to process in your central office. Imagine a retailer that needs to send information from every branch to the central. - Tweets analysis. Imagine you need to get information from different Twitter channels - Database tables - Etc. As we’ve depicted in the diagram, the minimum unit of configuration is a partition that belongs to a Topic, and Topics belong to Brokers. Topics can be replicated along with the Brokers. And Brokers can act in a cluster fashion. ## Architecture description Messages are published to Topics’ partitions and marked with an offset, which will be very useful for consumers at the time they start to read from the stream. ![Source: https://kafka.apache.org/documentation.html](../../assets/blog/3-scenarios-to-consume-an-apache-kafka-topic-with-mulesoft-3.png) Kafka is a very powerful platform, and part of that is that it works pretty smart when dealing with the idea of multiple partitions for a Topic. One of the main Kafka capabilities is the resiliency of the platform as well as the flexibility to read the messages even when one of the brokers may be experiencing an issue, or a consumer is having a problem. As you can see -in the previous image- a message is published into a Topic Partition (as a producer you can decide where to point to) and an offset is generated to mark the messages, which will help us when we want to read the messages. You can have single or multiple producers, producing messages on the same topic and the same or different partitions. It will depend on your design. But let’s not just leave it as a generic idea of “depends on your design,” which is true but this type of article has the objective to help you decide on how to use things, and that’s what we are going to do. This idea of having multiple producers producing messages on the same Topic but under different partitions can be to divide how the consumers will read the information. Let’s get back to the retail scenario where different branches need to send information to the central. Every branch has a group of PoS (Point-of-Sales). The Topic may represent the branch and partitions can represent every PoS. Then, from the central, we can have multiple consumers reading from the same Topic (branch) but pointing to different partitions (PoS), and processing the information individually. Maybe we need to differentiate every single PoS, and therefore we can assign consumers to read from that particular partition. Another idea is that a Topic represents a group of branches (regions for example), and every partition represents a single branch; you may have a specific consumer reading the information of a specific branch. As we’ve mentioned, it will depend on how you design it and that is fully related to your use case. ## Getting deep into the consumers' mechanics Let’s think about it from the consumer perspective and let’s continue elaborating on the retail scenario. If we think of a large chain of supermarkets with thousands of branches around the country, and the need is to process every single ticket that is generated by every single branch, and ultimately send it to the central, then the volume of messages will be something relevant in this system, and also the rhythm of consuming them. With a single consumer reading the messages from a single Topic and partition, may not be the most scalable system. But if we can have multiple consumers reading from different partitions, then we can have parallel multiple consumers. Kafka also offers a way for the consumers to be rebalanced automatically. That is very useful functionality, where we have a Topic with different partitions that are being fed by multiple producers, but from the consumer standpoint, we can have a group that could be pointing just to the Topic name, and Kafka will assign which partitions the different consumer are going to read from. And even better, in case a consumer goes down, Kafka will automatically rebalance the partitions across the available consumers. Ain’t that cool? ![Source: https://www.confluent.io/blog/cooperative-rebalancing-in-kafka-streams-consumer-ksqldb/](../../assets/blog/3-scenarios-to-consume-an-apache-kafka-topic-with-mulesoft-4.png) That is one of the main differences between Kafka and other messaging systems. For example, the good old JMS does offer different options for consumers and producers to consume and generate -respectively- messages. It can be a queue or a topic. For a queue, just one single consumer can be subscribed to it. For a topic, many consumers can read from the same Topic but will read the same messages. > [!NOTE] > We are not saying nor comparing Kafka with JMS right here, but what we are doing is highlighting the functionalities that Kafka brings to the table. But you may be asking yourself: ain’t this a MuleSoft article? Where is MuleSoft within all this contextual Kafka explanation? Well, it happens that MuleSoft offers a connector that can produce and consume messages from Topics. This article is not to explain Kafka but to talk about the usage of MuleSoft Apache Connector ([https://docs.mulesoft.com/kafka-connector/4.6/](https://docs.mulesoft.com/kafka-connector/4.6/)) and how you can have different options to read information from Kafka Topics. Apache Kafka offers different SDKs for your application to reduce the complexity to produce/reading information into a Topic/Partition. For example Scala, JAVA, .NET, Go, Python, etc. The connector is implementing those very same SDKs, so you can do most of the things that you can achieve through the SDKs, with the MuleSoft connector. ## Scenarios of MuleSoft and Kafka getting together If you are looking to use MuleSoft to read from Kafka, my suggestions are to first understand Kafka's mechanics before trying to configure the connector. The connector is very straightforward to use; but if you do not have a good design and understanding of Kafka, and you just configure a MuleSoft flow to consume messages and for example, just point it to a specific partition, you have a scenario where thousands of messages need to be consumed and the result of that is that you are consuming message on a good rhythm but not in the one you expect, then it is because you may not have all the context of Kafka. Or imagine that you have a cluster of MuleSoft Runtimes running on-premise and you have the need to connect to Kafka and you did a similar configuration as explained in the previous paragraph. You deploy your MuleSoft application into the cluster, and you realize that the message is being duplicated (you have a two-node MuleSoft cluster). Why is that happening? Is that what I wanted? Why is MuleSoft duplicating messages? Then you start doubting that your MuleSoft application is not working as you were expecting. You even go further and raise a ticket with Support. But you know why? The problem is not MuleSoft, it is a lack of understanding of how a MuleSoft cluster works and also how Kafka works. This article is going to explain the mechanisms that you can use within a MuleSoft application to read messages from a Topic and partitions, and which are the expected behaviors and consequences. We will go through these scenarios: - Using a configuration where you can point to different topics and/or partitions. Here you explicitly configure the adapter to do so. - Using a configuration where you just point to the Topic name without specifying a partition number. - Using the configuration within a MuleSoft cluster (runtime on-premise, hybrid mode). In all those cases, MuleSoft is behaving like a consumer. And maybe it is obvious, but we need to understand that the connector is going to behave like a normal Kafka consumer. Is nothing extra, as we’ve mentioned, it is built on top of the official Kafka SDKs, it is nothing more than that. That is why we need to understand Kafka to configure it properly and understand how to read the messages. Let’s go scenario by scenario. ## Scenario [#1](https://www.prostdev.com/blog/hashtags/1) - Connect to a specific Topic and Partition In this scenario, we will connect Kafka directly to a partition number and Topic. Why would we do that? Your use case may be one where you need to have specific MuleSoft applications pointing to specific combinations of Topics and Partitions. Let’s say it in another way: deliberately you have a MuleSoft application reading from a specific partition, period. You understand that in this scenario you will have a single application and a single flow reading from the partition, you may not want to have more than one application and/or flow connected to the same partition unless you want to duplicate your messages and that is not a problem for you. A very simple MuleSoft flow will be something like this: ![Anypoint Studio flow: a Kafka Message listener connected to a Logger](../../assets/blog/3-scenarios-to-consume-an-apache-kafka-topic-with-mulesoft-5.png) In this case, the configuration from the Kafka connector perspective is as follows: ![Kafka Consumer config with Assignments set to Edit inline and TopicA partition 1 specified](../../assets/blog/3-scenarios-to-consume-an-apache-kafka-topic-with-mulesoft-6.png) In this scenario, our MuleSoft connector configuration in the Topics section needs to be Assignments (orange box), if you select that one the next section (red box) will let you decide which Topic and Partition you want to read from. You need to specify those two values, there is no way to continue with the rest of the configuration unless you input the Topic name and partition number. What happens if I add another flow or even more, another MuleSoft application with the same configuration? Like this: ![Two Kafka consumer flows with identical Message listeners reading the same topic partition](../../assets/blog/3-scenarios-to-consume-an-apache-kafka-topic-with-mulesoft-7.png) Here we are telling Kafka that we have two consumers wanting to read whatever messages are in the Partition 1 of Topic TopicA. The result: **both flows are going to get triggered and will process (duplicating) all messages**. As we’ve mentioned it can be another flow or another application, but that configuration will produce that effect in terms of duplicating the messages. If your use case is just to read the message once and process it once, then you need just one single application and/or flow consuming from Kafka. If your use case is to read the same message multiple times and process them distinctively, then this configuration is also going to work for you. Maybe you want to perform different transformations to the same message and process it differently, then you can have a separate flow doing that. Another possible question you may have is: what happens with this type of configuration but deploying the application on a MuleSoft cluster or in Cloudhub with multiple replicas? That question will be answered in the Scenario [#3](https://www.prostdev.com/blog/hashtags/3) section. ## Scenario [#2](https://www.prostdev.com/blog/hashtags/2) - Connect to a Topic but without specifying the Partition Contrary to the previous scenario, in this case, we want to point just to the Topic and let Kafka balance the consumers to read from the multiple partitions. This is what we explained at the beginning of the article. Let’s think that we have a Topic with 8 partitions, and hundreds of thousands of messages are getting produced and ready to be consumed by a group of consumers (MuleSoft applications). Those messages are distributed across all the 8 partitions. If we used the strategy of the previous scenario -all consumers pointing to the same partition- we will not read all the messages in the first place, and also the consumption rhythm to get all those hundreds of thousand messages with a single consumer will not be the best idea. Let’s also think that in the future those messages may increase; if we think in the Retail scenario that we’ve been describing in previous sections of this article, a new branch can be introduced to the system and this will increase the number of messages as well as partitions. That can cause us to increase the number of consumers; but this needs to be as smooth as possible and as simple as incrementing them, without the need to modify our code of change configurations. For example, in CloudHub, we can simply increase the number of replicas of our application and that will increase the number of consumers. Kafka will see that a new consumer is registered and will assign it a partition on the fly. To use this type of configuration, we need to use the following: ![Kafka Consumer config using Topic Subscription Patterns set to TopicA with no partition](../../assets/blog/3-scenarios-to-consume-an-apache-kafka-topic-with-mulesoft-8.png) The orange box has the Topic subscription configuration, in this case: **Topic Subscription Patterns**. And the red box will appear after selecting this option and will ask you for a Topic Name, in our case: TopicA. You do not need to specify the partition; in this case, Kafka will balance the partition across the different MuleSoft applications that are acting as consumers. In this case, if we have something like the following: ![Two Kafka consumer flows with topic-subscription Message listeners that each read different partitions](../../assets/blog/3-scenarios-to-consume-an-apache-kafka-topic-with-mulesoft-9.png) We have two different MuleSoft flows with the same Message Listener configuration (the one we described in previous paragraphs), in this case, both flows will consume different messages from different partitions. Or it can also be the case of having different multiple MuleSoft applications with the same Message Listener configuration, and every application will consume different messages. If one of our applications goes down for any reason, Kafka will rebalance the partitions through the rest of the consumers. Ain’t that cool? This scenario is very powerful, you can mix Kafka’s capabilities with MuleSoft’s. Kafka with the consumer balancing, and MuleSoft with its scalability. Now let’s move to the 3rd scenario where we will see how this configuration and the previous one (Scenario [#1](https://www.prostdev.com/blog/hashtags/1)) are affected within a MuleSoft cluster. ## Scenario [#3](https://www.prostdev.com/blog/hashtags/3) - MuleSoft acting as a cluster and reading from Kafka Both of the previous scenarios will respond to questions regarding your use cases. Is not that one scenario is better than the other. There will be some consequences if we have a use case that does not fit with the way you configure the connector, and we do not want that to happen. We want this type of article, to give you suggestions and ideas on how to determine the best way to use MuleSoft and avoid surprises. Let’s imagine we have a four-node MuleSoft cluster. In this case; hybrid mode. Within a MuleSoft cluster, certain dynamics happen behind the scenes for us, but that we need to understand to avoid misunderstandings. For example: - If you have a MuleSoft application that contains a scheduler, the scheduler will only run on the master node, to avoid that it will trigger in all the nodes. - If you have a MuleSoft application that is using the Object Store, this object store is replicated along with the cluster nodes. This is pretty good since all members have access to the same information - VM Queues. They are also distributed through the different nodes, and in case of a failure, if you have it configured with persistency, the information in the VM Queue will resist node failures and will continue working on the rest of the nodes - If you are subscribed to a JMS queue, for example, and you are consuming from Queue, you will need that only the master node can consume messages, and in case of a failure, the next eligible master node will continue consuming them But in our case with Kafka? As we’ve described in the previous two scenarios, sometimes we need to connect to a specific partition or sometimes we just want to connect to a Topic and have Kafka distribute the partitions for us. In the first scenario, again, if the main purpose is to point to a specific partition. If you deploy that on a cluster without checking the following box: ![Message listener Advanced settings with the Primary node only checkbox highlighted](../../assets/blog/3-scenarios-to-consume-an-apache-kafka-topic-with-mulesoft-10.png) The expected behavior is that every single node that is part of the cluster and that has our application deployed is going to consume the same message. If we don’t want duplicated messages, then check the Primary node only box, and that’s it. Consequences of this: only the master node will read from the partition, and the rhythm of consumption will be determined by the power of that node and application. Also, in case of a failure of the master node, and once a new master node is elected, it will be the one responsible for consuming the messages. Scenario [#2](https://www.prostdev.com/blog/hashtags/2) is a different story. Here we **do not need** to use this checkbox, since the main point of using this configuration model is to have multiple consumers reading messages. If you keep the Primary node only checkbox selected, then you will have just the master node reading messages. The consequences of this are: - You will not read messages for all partitions, since the worker node will be pointing to the partition that Kafka determined when it was subscribed. - You lost horsepower since you will have the rest of the nodes without reading messages. The conclusion: for use cases that are more aligned to Scenario [#2](https://www.prostdev.com/blog/hashtags/2), do not use the Primary node only option, keep it disabled ## Conclusions MuleSoft and Kafka working together is a very powerful solution. Implementing them together, as in any other technology, imply that you need to know how both work and which are the different alternatives to mix them. In this article, we just talked about consumption, which in our opinion is the scenario with the most alternatives. Before going directly to Anypoint Studio to create your applications, we suggest you first analyze and study the solution as a whole, to get the most out of MuleSoft and the applications/platforms that it will integrate. --- ## Amazon DynamoDB Connector Operations in Mule 4 (Part 1) Source: https://prostdev.com/post/amazon-dynamodb-connector-operations-in-mule-4-part-1 | Published: Mar 29, 2022 | Category: Tutorials This blog post provides quick examples of how to use the Amazon DynamoDB Connector in Mule 4 using Anypoint Studio. There is a lot of documentation available on this connector’s usage, but none could be found that specifically shows how to structure the JSON requests, which are required for all DynamoDB connector operations. This article explains all the connector operations along with their JSON requests, with simple use case scenarios. We have considered all available DynamoDB data types for this use case, so you will have a very good understanding and usage of different DynamoDB data types. ## Operations Below is the list of connector operations considered for the first part of this series. - Create Table - Describe Table - List Tables - Put Item - Batch Put Item - Single Table - Batch Put Item - Multiple Tables - Get Item - Batch Get Item - Single Table - Batch Get Item - Multiple Tables - Update Item - Query - Scan - Delete Item - Batch Delete Item - Single Table - Batch Delete Item - Multiple Tables ## Use Case To better understand the DynamoDB operations, we have considered a use case of the *continents* & their *countries*. All this data is segregated based on types of information & stored into two DynamoDB tables. Each *continent* consists of multiple *countries*, & each *country* will have its own specific information like unique id, available city name, specialties, available zip codes, year-wise populations, etc. All this information is stored in the table `CONTINENT-COUNTRY-INFO-TABLE`. Along with the above information, countries also have some classified information, like available agents in-country and intelligence agency information. All this classified information is stored in the table `COUNTRY-CLASIFIED-INFO-TABLE`. This use case has taken into consideration *(almost)* all available DynamoDB connector operations and their data types for clear understanding. **CONTINENT-COUNTRY-INFO-TABLE** | **Column Name** | **Data Type** | **Attributes** | |---|---|---| | continent_name | String | primary-partition-key | | country_unique_id | Number | primary-sort-key | | country_name | String | | | isActive | Boolean | | | country_specialities | StringSet | | | country_available_pincodes | NumberSet | | | country_yearwise_population | Map | | | country_city_name | List | | | country_geo_value | null | | **COUNTRY-CLASIFIED-INFO-TABLE** | **Column Name** | **Data Type** | **Attributes** | |---|---|---| | country_unique_id | Number | primary-partition-key | | country_agents | Number | | | country_intelligence_agency | String | | ## 1. Create Table ![Mule flow: Listener, Create table DynamoDB operation, then Set Payload.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-1.png) Let's start with our first operation. This operation will create a new table with the provided configurations. > [!NOTE] > Execute this connector operation separately for both tables, with their own configurations. ![Create table config with continent_name HASH and country_unique_id RANGE keys and 100 read/write units.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-2.png) Below are the provided parameters: - **Table name:** The name for the table. - **AttributeDefinitions:** This value is an array of attributes that describe the key schema for the table and indexes. We have configured a composite primary key, which is a combination of primary key and sort key. - `continent_name`: Primary Partition Key - `country_unique_id`: Primary Sort key A composite primary key gives you additional flexibility when querying data. For example, if you provide only the value for `continent_name`, DynamoDB retrieves all of the `country_unique_id` associated with it. - **KeySchema**: This value specifies the attributes that make up the primary key for a table or an index. - **Read Capacity Units**: 100 – The maximum number of strongly consistent reads per second. - **Write Capacity Units**: 100 – The maximum number of writes consumed per second. **Response** Once you execute this connector, the DynamoDB tables will be created and you will receive a SUCCESS Response. ## 2. Describe Table ![Mule flow: Listener, Describe table DynamoDB operation, then Set Payload.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-3.png) This operation is used to get information about the table, including the current status of the table, when it was created, the primary key schema, and any indexes on the table. ![Describe table config with Table name set to CONTINENT-COUNTRY-INFO-TABLE.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-4.png) The only parameter we configure for this operation is the *Table name*. **Response** Once executed, the connector will return a response in the below format. ```json { "globalSecondaryIndexes": null, "latestStreamLabel": null, "latestStreamArn": null, "keySchema": [ { "attributeName": "continent_name", "keyType": "HASH" }, { "attributeName": "country_unique_id", "keyType": "RANGE" } ], "provisionedThroughput": { "writeCapacityUnits": 100, "readCapacityUnits": 100, "numberOfDecreasesToday": 0, "lastIncreaseDateTime": null, "lastDecreaseDateTime": null }, "tableStatus": "ACTIVE", "itemCount": 0, "tableArn": "arn:aws:dynamodb:xx-xxxx-x:xxxxxxxxxxxxxxxxx:table/CONTINENT-COUNTRY-INFO-TABLE", "tableSizeBytes": 0, "tableName": "CONTINENT-COUNTRY-INFO-TABLE", "streamSpecification": null, "attributeDefinitions": [ { "attributeType": "STRING", "attributeName": "continent_name" }, { "attributeType": "NUMBER", "attributeName": "country_unique_id" } ], "localSecondaryIndexes": null, "creationDateTime": "2022-03-07T11:13:56.413" } ``` ## 3. List Tables ![Mule flow: Listener, List tables DynamoDB operation, then Set Payload.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-5.png) This operation will return all the table names associated with your current AWS account and endpoint. ![List tables config with an Exclusive start table name and a Limit of 100.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-6.png) Below parameters are configured for this operation: - **Exclusive start table name**: The first table name that this operation will evaluate. If no table names are provided, it will list all tables. - **Limit**: A maximum number of table names to return. If this parameter is not specified, the limit is 100. **Response** Once executed, the connector will return a response in the below format. ```json { “tableNames” : [ “table-1”, “table-2”, “table-3” ], “lastEvaluatedTableName” : “table-4” } ``` ## 4. Put Item ![Mule flow: Listener, Transform Message for DynamoDB structure, Put item operation, then Set Payload.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-7.png) Use this operation when the need is to create a new item or replace an existing item in the DB. If an item that has the same primary key as a new item already exists in a specified table, the new item replaces an existing item. ![Put item config with Table name CONTINENT-COUNTRY-INFO-TABLE and Item set to payload.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-8.png) **Request** Let's now create our first country “India” with other required information. ```dataweave %dw 2.0 output application/json --- { "continent_name": { "S": "ASIA" }, "country_unique_id": { "N": 1 }, "country_name": { "S": "India" }, "isActive": { "Bool": true }, "country_specialities": { "ss": ["India is democratic country", "India has 121 languages and 270 mother tongues", "Jana-gana-mana is a national song of india"] }, "country_available_pincodes": { "ns": [110001, 110002, 110003] }, "country_yearwise_population": { "M": { "2020": { "N": "1380004385" }, "2021": { "N": "1393409038" }, "2022": { "N": "1406631776" } } }, "country_city_name": { "L": [ { "S": "Mumbai" }, { "S": "Delhi" }, { "S": "Pune" }, { "S": "Kolkata" } ] } } ``` **Response** Once you execute the above flow, you will see 1 record is created in the table `CONTINENT-COUNTRY-INFO-TABLE`. ## 5. Batch Put Item - Single Table ![Mule flow: Listener, Transform Message, Batch put item operation, then Set Payload.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-9.png) This operation inserts/puts multiple records or items in one table. **Request** Let's now create 2 country records “Singapore” and “Japan” in the same table `CONTINENT-COUNTRY-INFO-TABLE`. ```dataweave { "CONTINENT-COUNTRY-INFO-TABLE": [{ //Table1 "PutRequest": { // Item 1 "continent_name": { "S": "ASIA" } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_unique_id": { "N": 2 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_name": { "S": "Singapore" } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "isActive": { "Bool": true } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_specialities": { "ss": [ "Singapore is Renowned for having some of the cleanest streets in the world", "Singapore has a Reputation as the Garden City", "Singapore has Good Street Food"] } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_available_pincodes": { "ns": [1,2,3] } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_yearwise_population": { "M": { "2020": { "N": 5850342 }, "2021": { "N": 5896686 }, "2022": { "N": 5943546 } }} as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_city_name": { "L": [{ "S": "Central Region" }, { "S": "East Region" }, { "S": "North Region" }, { "S": "North-East Region" }, { "S": "West Region" }] } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" } } } as Object { class:"org.mule.extension.dynamodb.api.model.WriteRequest" }, { "PutRequest": { // Item 2 "continent_name": { "S": "ASIA" } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_unique_id": { "N": 3 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_name": { "S": "Japan" } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "isActive": { "Bool": true } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_specialities": { "ss": [ "Mount Fuji is the highest volcano in Japan", "Tokyo Skytree is the Tallest tower not just in Japan, but in the entire world", "Japan has the Fastest bullet train", "Geisha are a big part of Japanese tradition and culture"] } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_available_pincodes": { "ns": [1200000,1900100,1120000] } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_yearwise_population": { "M": { "2020": { "N": 126476461 }, "2021": { "N": 126050804 }, "2022": { "N": 125584838 } } } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_city_name": { "L": [{ "S": "Tokyo" }, { "S": "Osaka" }, { "S": "Kawasaki" }, { "S": "Chiba" }] } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" } } } as Object { class:"org.mule.extension.dynamodb.api.model.WriteRequest" }] } ``` **Response** Once you execute the above flow, you will see both these records are created in table `CONTINENT-COUNTRY-INFO-TABLE`. ## 6. Batch Put Item - Multiple Tables ![Mule flow: Listener, Transform Message, Batch put item - multiple table operation, then Set Payload.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-10.png) This operation Inserts/Puts multiple records or items in one or more tables. ![Batch put item config with Request put items set to the Expression #[payload].](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-11.png) **Request** Let's now create one more country “Switzerland” in table `CONTINENT-COUNTRY-INFO-TABLE` and all 4 countries classified information (India, Singapore, Japan, Switzerland) in table `COUNTRY-CLASIFIED-INFO-TABLE`. ```dataweave { "CONTINENT-COUNTRY-INFO-TABLE": [{ //Table1 "PutRequest": { // Item 1 "continent_name": { "S": "Europe" } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_unique_id": { "N": 4 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_name": { "S": "Switzerland" } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "isActive": { "Bool": true } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_specialities": { "ss": [ "Switzerland is famous for their watches", "Switzerland is a beautiful, tourist-attracting country", "Switzerland is famous for its mesmerizing alpine scenery, luxury branded watches, and deliciously milky chocolate"] } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_available_pincodes": { "ns": [1715,1732,1673] } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_yearwise_population": { "M": { "2020": { "N": 8654622 }, "2021": { "N": 8715494 }, "2022": { "N": 8773637 } }} as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_city_name": { "L": [{ "S": "Zurich" }, { "S": "Lucerne" }, { "S": "Basel" }, { "S": "Bern" }, { "S": "Geneva" }] } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" } } } as Object { class:"org.mule.extension.dynamodb.api.model.WriteRequest" }], "COUNTRY-CLASIFIED-INFO-TABLE": [{ "PutRequest": { "country_unique_id": { "N": 1 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_agents": { "N": 500000 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_intelligence_agency": { "S": "Research and Analysis Wing (RAW)" } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" } } } as Object { class:"org.mule.extension.dynamodb.api.model.WriteRequest" },{ "PutRequest": { "country_unique_id": { "N": 2 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_agents": { "N": 120000 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_intelligence_agency": { "S": "The Security and Intelligence Division (SID)" } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" } } } as Object { class:"org.mule.extension.dynamodb.api.model.WriteRequest" },{ "PutRequest": { "country_unique_id": { "N": 3 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_agents": { "N": 90000 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_intelligence_agency": { "S": "The Public Security Intelligence Agency" } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" } } } as Object { class:"org.mule.extension.dynamodb.api.model.WriteRequest" }, { "PutRequest": { "country_unique_id": { "N": 4 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_agents": { "N": 50000 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_intelligence_agency": { "S": "The Swiss Intelligence Agency" } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" } } } as Object { class:"org.mule.extension.dynamodb.api.model.WriteRequest" } ] } ``` **Response** Once you execute the above flow, you will see all these records are created in the respective table. ## 7. Get Item ![Get item flow with a Transform Message building the key with continent_name ASIA and country_unique_id 2.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-12.png) The GetItem operation returns a set of attributes for the item with the given primary key. If there is no matching item GetItem does not return any data. ![Get item config with Table name CONTINENT-COUNTRY-INFO-TABLE, Key set to payload, and Consistent read False.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-13.png) **Request** Let's now fetch the record from table `CONTINENT-COUNTRY-INFO-TABLE`, having continent = ‘ASIA’ & country unique id = ‘2’. ```dataweave %dw 2.0 output application/json --- { "continent_name" : {"S" : "ASIA"}, "country_unique_id" : {"N" : 2} } ``` **Response** Once you execute the above flow, you will see the respective records in response. ## 8. Batch Get Item - Single Table ![Mule flow: Listener, Transform Message, Batch get item - single table operation, then Set Payload.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-14.png) The Batch Get Item operation returns the attributes of one or more items from one table. You identify the requested item by primary key and sort key combination. ![Batch get item config with Request items set to the Expression #[payload] and Return consumed capacity TOTAL.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-15.png) **Request** Let’s now fetch two records from table `CONTINENT-COUNTRY-INFO-TABLE` , having continent = ‘ASIA’ & ‘Europe’ & country unique id = ‘1’ & ‘4’. ```dataweave { "CONTINENT-COUNTRY-INFO-TABLE" : { "Keys": [ { "continent_name" : {"S":"ASIA"} as Object {class:"org.mule.extension.dynamodb.api.model.AttributeValue"}, "country_unique_id":{"N": 1} as Object {class:"org.mule.extension.dynamodb.api.model.AttributeValue"} }, { "continent_name" : {"S":"Europe"} as Object {class:"org.mule.extension.dynamodb.api.model.AttributeValue"}, "country_unique_id":{"N": 4} as Object {class:"org.mule.extension.dynamodb.api.model.AttributeValue"} } ] } as Object {class:"org.mule.extension.dynamodb.api.model.KeysAndAttributes"} } ``` **Response** Once you execute the above flow, you will see both these respective records in response. ## 9. Batch Get Item - Multiple Tables ![Mule flow: Listener, Transform Message, Batch get item - multiple table operation, then Set Payload.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-16.png) The Batch Get Item operation returns the attributes of one or more items from one or more tables. You identify the requested item by primary key. ![Batch get item config for multiple tables with Request items #[payload] and Return consumed capacity TOTAL.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-17.png) **Request** Let's now fetch two records each from table `CONTINENT-COUNTRY-INFO-TABLE` & `COUNTRY-CLASIFIED-INFO-TABLE`. ```dataweave { "CONTINENT-COUNTRY-INFO-TABLE": { "Keys": [{ "continent_name": { "S": "ASIA" } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_unique_id": { "N": 1 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" } }, { "continent_name": { "S": "ASIA" } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_unique_id": { "N": 2 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" } }] } as Object { class:"org.mule.extension.dynamodb.api.model.KeysAndAttributes" }, "COUNTRY-CLASIFIED-INFO-TABLE": { "Keys": [{ "country_unique_id": { "N": 1 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" } }, { "country_unique_id": { "N": 2 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" } }] } as Object { class:"org.mule.extension.dynamodb.api.model.KeysAndAttributes" } } ``` **Response** Once you execute the above flow, you will see both these respective records from each table, in the response. ## 10. Update Item ![Update item flow with a Transform Message building the key for continent_name ASIA and country_unique_id 1.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-18.png) This operation edits an existing item's attributes or adds a new item to the table if it does not already exist. ![Update item config with a SET update expression and inline attribute values for isActive and countryName.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-19.png) **Request** Let's now update the record with country name "India" with: - `isActive = 'false'` - `country_name = 'India/Bharat'` ```dataweave %dw 2.0 output application/json --- { "continent_name": {"S": "ASIA"} as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_unique_id": {"N": 1} as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" } } ``` **Response** Once the above request is served, both these columns (`isActive` & `country_name`) will be updated respectively with the new values. ## 11. Query ![Query flow with a Transform Message building criteria continent_name ASIA and isActive true.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-20.png) The Query operation finds items based on primary key values. You can query any table that has or secondary index that has a composite primary key (a partition key and a sort key). ![Query config with the key condition expression continent_name = :continent_name on the partition key.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-21.png) ![Query parameters mapping #isActive and the :continent_name and :isActive attribute values from the payload.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-22.png) ![Query config with filter expression isActive = :isActive and a Limit of 10.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-23.png) **Request** Now, Let’s query the table `CONTINENT-COUNTRY-INFO-TABLE` and fetch all the records that satisfy the following criteria: - `continent_name = 'ASIA'` - `isActive = 'true'` ```dataweave %dw 2.0 output application/json --- { "continent_name" : {"S" : "ASIA"}, "isActive" : {"Bool" : true} } ``` **Response** Once you execute the above flow, you will find 3 records in the response. ## 12. Scan ![Scan flow with a Transform Message building criteria continent_name Europe and isActive true.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-24.png) The scan operation returns one or more items or item attributes By accessing every item in a table or secondary index. To have DynamoDB return fewer items, you can provide a FilterExpression operation. ![Scan config with Segment 0, Total segments 1, and continent_name and isActive attribute values.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-25.png) ![Scan config with two filter expressions on continent_name and isActive and a Limit of 100.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-26.png) **Request** Let's scan the table `CONTINENT-COUNTRY-INFO-TABLE` and fetch all the records that satisfy the following criteria: - `continent_name = 'Europe'` - `isActive = 'true'` ```dataweave %dw 2.0 output application/json --- { "continent_name" : {"S" : "Europe"}, "isActive" : {"Bool" : true} } ``` **Response** Once you execute the above flow, you will find only 1 record in response. ## 13. Delete Item ![Mule flow: Listener, Transform Message single delete request, Delete item operation, then Set Payload.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-27.png) This operation deletes a single item in a table. Items are identified using primary key and sort key combination. ![Delete item flow with a Transform Message building the key continent_name ASIA and country_unique_id 3.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-28.png) **Request** Let's now delete the record with country = ‘Japan’ from table `CONTINENT-COUNTRY-INFO-TABLE` ```dataweave %dw 2.0 output application/json --- { "continent_name": { "S": "ASIA" } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_unique_id": { "N": 3 } as Object { class:"org.mule.extension.dynamodb.api.model.AttributeValue" } } ``` **Response** Once executed, the record will be deleted from the table. ## 14. Batch Delete Item - Single Table ![Mule flow: Listener, Transform Message, Batch delete item - single table operation, then Set Payload.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-29.png) This operation deletes multiple items in one table. ![Batch delete item config with Request delete items set to the payload expression.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-30.png) **Request** We’ll now delete 2 records in a single request. Let’s delete both countries “Singapore” & “Switzerland” from the table `CONTINENT-COUNTRY-INFO-TABLE`. ```dataweave { "CONTINENT-COUNTRY-INFO-TABLE": [{ "DeleteRequest": { "continent_name": { "S": "ASIA" } as Object { class: "org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_unique_id": { "N": "2" } as Object { class: "org.mule.extension.dynamodb.api.model.AttributeValue" } } } as Object { class : "org.mule.extension.dynamodb.api.model.WriteRequest" }, { "DeleteRequest": { "continent_name": { "S": "Europe" } as Object { class: "org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_unique_id": { "N": "4" } as Object { class: "org.mule.extension.dynamodb.api.model.AttributeValue" } } } as Object { class : "org.mule.extension.dynamodb.api.model.WriteRequest" }] } ``` **Response** Once executed, both these records will have been deleted from the table. ## 15. Batch Delete Item - Multiple Tables ![Mule flow: Listener, Transform Message, Batch delete item - multiple table operation, then Set Payload.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-31.png) This operation deletes multiple items from multiple tables. ![Batch delete item config for multiple tables with Request delete items set to the payload expression.](../../assets/blog/amazon-dynamodb-connector-operations-in-mule-4-part-1-32.png) **Request** We’ll delete country ‘India’ from table `CONTINENT-COUNTRY-INFO-TABLE` and its classified information from the table `COUNTRY-CLASIFIED-INFO-TABLE`. ```dataweave { "CONTINENT-COUNTRY-INFO-TABLE": [{ "DeleteRequest": { "continent_name": { "S": "ASIA" } as Object { class: "org.mule.extension.dynamodb.api.model.AttributeValue" }, "country_unique_id": { "N": 1 } as Object { class: "org.mule.extension.dynamodb.api.model.AttributeValue" } } } as Object { class : "org.mule.extension.dynamodb.api.model.WriteRequest" } ], "COUNTRY-CLASIFIED-INFO-TABLE": [{ "DeleteRequest": { "country_unique_id": { "N": 1 } as Object { class: "org.mule.extension.dynamodb.api.model.AttributeValue" } } } as Object { class : "org.mule.extension.dynamodb.api.model.WriteRequest" }, { "DeleteRequest": { "country_unique_id": { "N": 3 } as Object { class: "org.mule.extension.dynamodb.api.model.AttributeValue" } } } as Object { class : "org.mule.extension.dynamodb.api.model.WriteRequest" }] } ``` **Response** Once executed, all 3 records will have been deleted from both tables. ## Conclusion We have seen the working examples of 15 operations of DynamoDB connectors. Stay tuned for the next part. Thank you and happy learning!! --- ## Guidelines to submit your content in ProstDev Source: https://prostdev.com/post/guidelines-to-submit-your-content-in-prostdev | Published: Mar 22, 2022 | Category: News Thank you for your interest in creating a post with us! We've been getting more articles lately but our bandwidth to review them is a bit decreased at the moment. Because of this, the time we spend reviewing the posts will need to be reduced. We want to keep serving our readers with great quality content, so our quality standards will have to be enforced by you **before** sending your application. Once you've followed the guidelines, we can start the review process. Your application might be **delayed** or **rejected** if your contribution does not comply with the standards. - Your article needs to be created in Google Docs before submitting. - Please make sure your document has the appropriate editing access for us to make editions to your file. - The title of your article must be 100 characters or less. - The contents of your article must be between 500 and 1,500 words. Note that specific content within the article (like tables or code blocks) might make the article longer than it is. If this is the case, please let us know when you submit your application or create gist files for the code (next point). - Any code blocks with more than 5 lines of code must be replaced with [public gist files](https://docs.github.com/en/get-started/writing-on-github/editing-and-sharing-content-with-gists/creating-gists). Please read this [Article Formatting section](https://www.prostdev.com/post/prostdev-s-new-blog-post-template#viewer-eg9k8) for more information. - If you copy/paste quotes, pictures, or code from other sites, you need to provide the link where you got it from. We don't accept any form of plagiarism. - We will reject content that has not been proofread before submission. To proofread your content, you can use services or systems that check your grammar, like [Grammarly](https://grammarly.com/) or [Hemingway](https://hemingwayapp.com/); or ask a friend to review your content first. - Once you've taken care of all these, you can submit your content [here](https://github.com/ProstDev/blog-posts/issues/new/choose). Please note that you will need a GitHub Account. Once you submit your content, we might take between 1-2 weeks to get back to you, depending on our current content queue. Any missed guideline/standard from the previous list will result in rejection or further delays in your review process. If your content is reviewed on time and follows the appropriate standards, we can publish your content as soon as 2 weeks. If further editions are required from your side, the publishing date will be extended. Any round of editions might add an additional week to the publishing date depending on our queue. Please be patient. Thank you! --- ## Anypoint Platform Single Sign-on (SSO) SAML Configuration with Oracle IDCS - PART 2 Source: https://prostdev.com/post/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2 | Published: Mar 15, 2022 | Category: Tutorials Our[last article](https://www.prostdev.com/post/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1) was about how to integrate MuleSoft Anypoint Platform with Oracle Identity Cloud Services. With this, you can use Oracle IDCS as your Identity Provider. This is very useful for Single Sign-On purposes, and if you already have Oracle Cloud Infrastructure and you are using MuleSoft to integrate it with Oracle SaaS apps, this alternative for SSO should be a good fit for you. In our last article, we mentioned that we would have a second part where we explain how to map attributes, such as - Email - First name and Last name - Telephone Number - Groups - Etc Your users are part of Oracle Identity Cloud Services. Some of those users need to enter to MuleSoft Anypoint Platform to perform different activities, like - Design APIs - Browse Exchange - Deploy Applications - Manage your organization But you don’t want to assign those roles directly on MuleSoft, you want to have that information coming from your Identity Provider; at the end that is the place where you create your users and assign groups. So, it is natural to think that the user is already assigned to groups that can be mapped to your MuleSoft roles. Once the users log in to MuleSoft, through the Identity Provider and it Authenticates/Authorizes them, and finally are able to get into MuleSoft, the expectation is that the users are already assigned to their roles and start working with their duties. Well, that is something that you can configure between MuleSoft and Oracle IDCS. Let’s get back to the IDCS console and create a group: ![Oracle IDCS Groups list with the Add button highlighted to create a new group](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-1.png) We just need to click on the Add button and create a group: ![Add Group step 1 naming the group CloudhubAdminSbx with a description](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-2.png) We are naming the user: CloudhubAdminSbx, and giving a brief description. Then click on Next and let’s assign a user: ![Add Group step 2 assigning the user Rolando Carrasco to the new group](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-3.png) In this case, it is me: [rcarrascogb2@me.com](mailto:rcarrascogb2@me.com). Then just click on Finish. What we have done is to create a new group (take a note of the name because we will use it later to map it inside MuleSoft Anypoint Platform), and we are assigning a user to it. Now let’s get back to our Application inside IDCS (the one we configured in our previous article), to map attributes and groups. Let’s do it. ![IDCS Mule application SSO Configuration page with Attribute Configuration highlighted](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-4.png) In the Attribute configuration, click on it and you will see something like this: ![IDCS Attribute Configuration table mapping email, firstname, lastname, and Groups](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-5.png) Your table may be empty and you will need to add the needed attributes, in my case: - email - firstname - lastname - Groups Those are the attributes that will be sent from IDCS to MuleSoft inside the SAML assertion and MuleSoft will map into its attributes. Take a look at the names, and get back to MuleSoft Anypoint Platform console to the Access Management menu: ![Anypoint Access Management Identity Providers list showing Anypoint and Oracle IDCS](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-6.png) We created this configuration in our previous [article](https://www.prostdev.com/post/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1), click on it. Go almost at the bottom of that page: ![Anypoint IdP advanced settings with first name, last name, email, and group attribute fields](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-7.png) Those names are mapped into MuleSoft and they need to match. In IDCS those fields are going to be filled with the user information. If we zoom in to the configuration at IDCS you will see this: ![IDCS attribute table with Name, Format, Type, and Value columns color-boxed for emphasis](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-8.png) In orange are the attributes that we want to return, in purple are the format in this case basic. In blue is the type of attribute and finally the value that IDCS will populate on them in red. And for the groups in specific we’ve configured it like this: ![Groups attribute set to Group Membership with the All Groups option selected](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-9.png) We are returning all the groups that the user is a member of. We have other options, like: ![Group filter dropdown open showing Equals, Starts with, and All Groups options](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-10.png) That means that our users can be assigned into multiple IDCS groups, but we can filter them to return just the ones the application needs. For this article, and to be practical, we are returning all of them. Everything is in place, our user ([rcarrascogb2@me.com](mailto:rcarrascogb2@me.com)) can try to log in to MuleSoft, and not only that will make it through, but at the same type, depending on the list of groups where the user belongs, it will map it into MuleSoft roles. But the question is: how does MuleSoft know that those group names (which by the way are arbitrary) are getting mapped with their roles? Just go to MuleSoft Anypoint Platform, and go to the Roles section: ![Anypoint Access Management Roles list including Cloudhub Admin (Sandbox)](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-11.png) Click on the group Cloud Admin (Sandbox) and you will see this: ![Cloudhub Admin (Sandbox) role page with the Set external group mapping link highlighted](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-12.png) Click on “Set external group mapping”, and you will see this: ![External IdP Groups dialog mapping group CloudhubAdminSbx to the Oracle IDCS provider](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-13.png) Type the group name that we’ve created at IDCS, choose the Identity Provider, and finally click on the add button. ![External IdP Groups dialog after the CloudhubAdminSbx Oracle IDCS mapping is added](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-14.png) Click on Save. Before testing it, let’s get back to MuleSoft Anypoint Platform, and review the user information: ![User Roles tab listing only the Cloudhub Admin (Design) role before login](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-15.png) The user has only that role assigned. But now, after his next login, it will have an additional role ([Cloudhub Admin (Sandbox)](https://anypoint.mulesoft.com/accounts/#/cs/core/roles/edit/66151060-bb95-499d-868f-bd4bebbb7c7d/permissions)). Also, as we saw in our previous article, the user did not have the attributes being mapped: ![Users list showing the Oracle IDCS user as "unknown unknown" with only an email mapped](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-16.png) No name (firstname and last name), just the email was mapped. Now the new configuration will populate that information as well as the group. Let’s see this action, let’s try to log in: ![Anypoint Platform sign-in page with a "Continue with Oracle IDCS" button](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-17.png) Click on Continue with Oracle IDCS: ![Oracle Cloud login page (in Spanish) with the user's email and password filled in](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-18.png) After we click on login, let’s capture the SAML response from IDCS: ![Captured SAML response XML with the firstname, lastname, email, and groups attributes](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-19.png) In pink, you can see that the firstname attribute was returned by Oracle IDCS. With orange (and highlighted in red), you can see the list of groups, including the one we created previously. And in blue we also see the lastname and email attributes being returned. Now, if we go to the Roles section of the user rcarrascogb2@me.com, we will see that the role has been assigned: ![User Roles tab now listing both Cloudhub Admin Sandbox and Design roles after login](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-2-20.png) With this, you can provision users into your Identity Provider, assign groups at that level and then map them into MuleSoft. I hope this is useful for you. --- ## Basic Google Big Query operations with a Salesforce sync demo in Mule 4 Source: https://prostdev.com/post/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4 | Published: Mar 1, 2022 | Category: Tutorials *GitHub repository with the Mule project can be found at the end of the post.* If we think about data storage the first thing that comes to our mind is a regular database, this can be any of the most popular ones like Mysql, SQL Server, Postgres, Vertica, etc. But I noticed not too many have interacted with one of the services Google provides with the same purpose **Google BigQuery.** And maybe it is because of the [pricing](https://cloud.google.com/bigquery/pricing?utm_source=google&utm_medium=cpc&utm_campaign=na-US-all-en-dr-bkws-all-all-trial-e-dr-1011347&utm_content=text-ad-none-any-DEV_c-CRE_573148306951-ADGP_Desk%20%7C%20BKWS%20-%20EXA%20%7C%20Txt%20~%20Data%20Analytics%20~%20BigQuery_Pricing%20Google%20Google-KWID_43700068582852990-kwd-166600832170&utm_term=KW_google%20bigquery%20pricing-ST_google%20bigquery%20pricing&gclsrc=aw.ds&gclid=Cj0KCQiAjJOQBhCkARIsAEKMtO3fCohKU2ihxQrQ21u9XixqWhXy9w7QNu8MqGaVp58zn59WTN0ekA4aAtAeEALw_wcB), but in the end, many companies are moving to cloud services and this service seems to be a great fit for them. In this post, I would like to demonstrate in a few steps how we can make a sync job that allows us to describe a Salesforce instance and use a few objects to create a full schema of those objects (tables) into a Google Big Query Dataset. Then with the schema created we should be able to push some data into BigQuery from Salesforce and see it in our Google Cloud Console project. ## Prerequisites To connect to Salesforce and Google BigQuery, there are a few prerequisites we need. **Salesforce:** - If you don’t have a salesforce instance, you can create a developer one [here](https://developer.salesforce.com/signup). - From the Salesforce side, you will need a **username, password,** and **security token** (you can follow [this process](https://help.salesforce.com/s/articleView?id=sf.user_security_token.htm&type=5) to get it). - A developer instance contains a few records, but if you need to have some more data, this will help the process to sync that information over. **GCP (Google Cloud Platform)** - You can sign up [here](https://console.cloud.google.com/freetrial/signup/tos?_ga=2.68094097.1640278748.1644510554-1516430238.1644510554&_gac=1.218077796.1644510554.Cj0KCQiAjJOQBhCkARIsAEKMtO0NyXkbcz86jMGZOta5V7HYUNkiDHCDR_6OSc4ioZFtAHlp0tw8_JUaAnI7EALw_wcB) for free. Google gives you $300 for 90 days to test the product (similar to Azure). If you already have a Google account, you can use it for this. ## Creating a new project in GCP and setting up our service account key Once you sign up for your account on GCP, you should be able to click on the New Project option and write a project name, in this example I chose **mulesoft.** ![GCP "Select a project" dialog with an arrow pointing to the New Project button](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-1.png) ![GCP New Project form with a project name field and Create button](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-2.png) Once a project is created we should be able to go to the menu on the left and we should be able to select IAM & Admin > Service Accounts option. ![GCP left menu with IAM & Admin expanded and Service Accounts highlighted](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-3.png) Now, we should be able to create our service account. “A **service account** is a special type of Google account intended to represent a non-human user that needs to authenticate and be authorized to access data in Google APIs. Typically, service accounts are used in scenarios such as Running workloads on virtual machines.” At the top of the page, you should be able to see the option to create it. Then, you just need to specify a Name and click on create and continue. ![Create service account form with a name field set to "sample"](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-4.png) The next step is to set the permissions, so for this, we need to select from the roles combo BigQuery Admin. ![Role selector dropdown with BigQuery Admin chosen for the service account](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-5.png) Once created, we should be able to select from the three-dot menu on the right the option **Manage Keys.** ![Service account three-dot Actions menu with Manage keys highlighted](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-6.png) Then we can create a new Key, in this case, one as JSON should be enough. The key will get downloaded automatically to your computer (**please keep this JSON key somewhere you can use it later**). ![Add Key dropdown showing Create new key and Upload existing key options](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-7.png) ### Dataset in Big Query Datasets are top-level containers that are used to organize and control access to your tables and views. A table or view must belong to a dataset, so you need to create at least one dataset before loading data into BigQuery. From the left menu, we can search for BigQuery and click on it. ![GCP navigation menu with BigQuery pinned under the Pinned section](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-8.png) That will take us to the BigQuery console. Now we can click on the three-dots menu and select the **Create dataset** option. ![BigQuery Explorer with the project's three-dot menu showing Create dataset](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-9.png) Now we just need to set the name as salesforce and click on “Create Dataset.” ![BigQuery Create dataset panel with Dataset ID set to "salesforce"](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-10.png) ## Setting up our Mule application Since this is a sync job, we don’t need any API specification but totally can fit some scenarios where we have another application that needs to consume specific endpoints/operations. Let’s then open our Anypoint Studio app (In my case I’m using a mac) and let’s use the default template. For this we are going to create six flows: - **Sync**. This flow just triggers the process. - **DescribeInstance.** This flow will be in charge of calling the describe operation using the Salesforce connector and provide all objects information from the Salesforce instance, also will have a loop that will allow us to process the job for the objects we are going to use. - **DescribeIndividualSalesforceObject.** Allows to describe a specific Salesforce object, this will basically will capture the fields and field types (STRING, EMAIL, ID, REFERENCE, etc.) and will be in charge to create a payload that BigQuery will recognize in order to get created in GBQ - **BigQueryCreateTable.** This flow only will be in charge of creating the table in BigQuery based on the Salesforce object name and the fields. - **QuerySalesforceObject.** This flow dynamically will query the Salesforce object and pull the data. (*For this we are limiting the output to 100 records but on a bigger scale it should be done on a batch process of course.*) - **InsertDataIntoBigQuery**. This flow will push the data over into BigQuery only. > [!NOTE] > I will go over in detail each of the flows in the next sections. Now let’s grab our JSON key generated by google and copy the file under the `src/main/resources` folder. The key will let us authenticate against our project and execute the operations. ![Studio package tree showing the JSON key under src/main/resources](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-11.png) ### Import the Google Big Query connector From Exchange, we can search “BigQuery” and we should be able to see the connector listed. ![Exchange listing for the Google BigQuery Connector for Mule 4](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-12.png) Then, we can just use the “Add to project” option and we should be able to see the operations in the Mule Palette. ![Mule Palette listing the Google BigQuery connector operations](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-13.png) ### Sync Flow As I mentioned, this is only in charge of triggering the whole application, so we only need one scheduler component and a flow reference to the DescribeInstance flow. ![Sync flow with a Scheduler connected to a DescribeInstance flow reference](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-14.png) ### DescribeInstance This flow will describe the whole Salesforce instance using the [Describe Global](https://developer.salesforce.com/docs/atlas.en-us.208.0.api_rest.meta/api_rest/resources_describeGlobal.htm) operation. The next step on this is to use a DataWeave transform to get only the Objects we are interested in. So in this case, I’m only pulling three: Accounts, Contacts, and a custom object called `Project__c`. I left in the transformation a few more attributes to only pull the objects that we can query. ```dataweave payload.sobjects filter ( $.queryable == true and $.replicateable == true and $.retrieveable == true and $.searchable == true and $.triggerable == true and ($.name == "Account" or $.name == "Contact" or $.name =="Project__c")) map $ ``` ![DescribeInstance flow: Describe global, a filter transform, then four flow references in a loop](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-15.png) Finally, you need to loop over these three objects and there’s a flow reference for this sample that will call the other flows to be able to continue the process. ### DescribeIndividualSalesforceObject This flow takes the name of the Salesforce Object and will allow it to describe it. The connector only asks for the object name, so we have a pretty interesting DW Transformation. ```dataweave %dw 2.0 input payload application/java output application/java fun validateField(field) = if ( (field == "REFERENCE") or (field == "ID") or (field == "PICKLIST") or (field == "TEXTAREA") or (field == "ADDRESS")or (field == "EMAIL")or (field == "PHONE") or (field == "URL")) "STRING" else if ( (field == "DOUBLE") or (field == "CURRENCY") ) "FLOAT" else if ((field == "INT")) "INTEGER" else field --- (payload.fields filter ($."type" != "LOCATION") map { fieldName : $.name, fieldType : validateField($."type") }) ``` Salesforce data types are not 100% the same as BigQuery, so we need to make a little trick to be able to create the schema in BigQuery seamlessly as Salesforce. In this case, I’ve created a small function to convert some fields (like ID, REFERENCE, TEXTAREA, PHONE, ADDRESS, PICKLIST, EMAIL) to be STRING. In this case, the reference or values are not really anything else than a text. For DOUBLE and CURRENCY, I’m using the value FLOAT. Finally, for INT fields are changed to be INTEGER. Because **Location** fields are a bit tricky and we are not able to make much with the API on them, I’m removing all location fields. The output of this is the actual schema we will use to create the table in Google BigQuery. ![DescribeIndividualSalesforceObject flow: Describe SObject then two Transform Message steps](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-16.png) ### BigQueryCreateTable This flow allows us to create the table in BigQuery; we only need to specify Table, Dataset, and Table Fields. ![Create Table operation config with Table from a variable and Dataset set to salesforce](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-17.png) ![BigQueryCreateTable flow with a single Create Table with Fields operation](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-18.png) ### QuerySalesforceObject This flow queries the Object in Salesforce and then maps the data dynamically to prepare the payload for BigQuery. ![Salesforce Query config with a dynamic SELECT and field/table parameters](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-19.png) The query comes from a variable “salesforceFields” the same field we collected when we described the Object using this script: ```dataweave (payload.fields filter ($."type" != "LOCATION") map { fieldName : $.name }).fieldName joinBy "," ``` And finally, I’m limiting the result to only 100 records. The next step is to map the Salesforce result data and map it dynamically using this script: ```dataweave %dw 2.0 import try, fail from dw::Runtime output application/java fun isDate(value: Any): Boolean = try(() -> value as Date).success fun getDate(value: Any): Date | Null | Any = ( if ( isDate(value) ) value as Date as String else value ) --- (payload map (item,index) ->{ (item mapObject ((value, key, index) -> { (key):(getDate(value)) } )) }) ``` Thanks so much to Alex Martinez for the insights on the utilities for DW 2.0! ([https://github.com/alexandramartinez/DataWeave-scripts/blob/main/utilities/utilities.dwl](https://github.com/alexandramartinez/DataWeave-scripts/blob/main/utilities/utilities.dwl)) This last script maps the records and uses the key as field and the value, but the value needs to be replaced as Date in this case for the Strings that are date or date-time. So I consider this the best script in this app. ![QuerySalesforceObject flow: Query, a Dynamic Field Mapping transform, then Logger](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-20.png) ### InsertDataIntoBigQuery This flow just inserts the data we prepared, so basically, we only need to specify table id, dataset id, and the Row Data. ![InsertDataIntoBigQuery flow with an Insert All operation followed by a Logger](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-21.png) ![Insert All operation config with Table id from a variable and Rows data set to payload](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-22.png) ## Setting up our Mule application Now we should be able to run our application and see the new tables and the data over Google BigQuery. On GCP, I can see the tables I selected being created: ![BigQuery Explorer showing the salesforce dataset with Account, Contact and Project__c tables](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-23.png) And if we open any of them we should look into the schema to verify all fields are there. ![BigQuery Contact table schema listing field names, types and nullable modes](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-24.png) Finally, we should be able to query the table in the console or click on the Preview option to check the data is there. ![BigQuery query results showing synced Account rows from Salesforce](../../assets/blog/basic-google-big-query-operations-with-a-salesforce-sync-demo-in-mule-4-25.png) I think this is kind of a common request we get on the integration space and many tweaks can be implemented if we are thinking of big migrations or setting some jobs that eventually will require tables to be created automatically from Salesforce to GCP. ## GitHub repository If you’d like to try it, I created this [GitHub](https://github.com/emoran/sfdc-to-bigquery) repository. I hope this was useful and I’m open to hearing any enhancement/scenario. --- ## Oracle database Stored Procedure does not time out as expected in MuleSoft Source: https://prostdev.com/post/oracle-database-stored-procedure-does-not-time-out-as-expected-in-mulesoft | Published: Feb 15, 2022 | Category: Guides In a recent MuleSoft project, I noticed a problem when trying to run an Oracle stored procedure that took a fair amount of time to complete. ## Context Even though the query timeout parameter was set in my code, I noticed that the execution of the MuleSoft flow kept waiting for the stored procedure to finish its execution and then continued the execution of subsequent processors in the flow. Here's an example flow I built to exemplify this behavior. I have an **HTTP listener** that accepts HTTP requests in the **/stored-proc** resource. The next processor is the **call to the stored procedure** in the Oracle database. This stored procedure does nothing but wait a random amount of time (from 5 to 10 seconds) and then write a message to the log. This stored procedure was intentionally built that way to simulate long-running execution. Finally, the flow **sets a success message** to the payload, and it is returned in the **HTTP response**. ![Mule flow: HTTP Listener, stored procedure call WAIT_FOR_A_WHILE, then Set Payload success](../../assets/blog/oracle-database-stored-procedure-does-not-time-out-as-expected-in-mulesoft-1.png) I am using a Docker container for the Oracle database. It is always much simpler to set up your environment using Docker containers to avoid installing all the software in a local environment. If you are interested in using this container, you can get it from DockerHub: [https://hub.docker.com/_/oracle-database-enterprise-edition](https://hub.docker.com/_/oracle-database-enterprise-edition) Or execute the following Docker command: ```bash docker pull store/oracle/database-enterprise:12.2.0.1 ``` ## Oracle Stored Procedure Below is the code I used to create the stored procedure in the database. As I mentioned earlier, the stored procedure only waits a few seconds before writing a message to the database log. ```sql CREATE OR REPLACE PROCEDURE ANONYMOUS.WAIT_FOR_A_WHILE IS BEGIN DBMS_LOCK.Sleep(DBMS_RANDOM.VALUE(5,10)); SYS.DBMS_OUTPUT.PUT_LINE(to_char(sysdate, 'hh24:mi:ss') || ': finished.'); END WAIT_FOR_A_WHILE; ``` ## MuleSoft flow configuration The processors that are part of the flow are configured as follows: 1. The HTTP listener listens on port 18081 and on the /stored-proc resource. ![HTTP Listener config with the path set to /stored-proc highlighted](../../assets/blog/oracle-database-stored-procedure-does-not-time-out-as-expected-in-mulesoft-2.png) 2. The database connector executes the stored procedure and has a configured timeout of 2 seconds. My expectation is that by taking more than 2 seconds, the connector returns an error due to timeout, and the flow execution is interrupted. ![Database connector calling WAIT_FOR_A_WHILE with a 2-second query timeout configured](../../assets/blog/oracle-database-stored-procedure-does-not-time-out-as-expected-in-mulesoft-3.png) 3. The set payload only places a message after successfully executing the stored procedure. ![Set Payload returning a JSON "Stored proc executed." success message](../../assets/blog/oracle-database-stored-procedure-does-not-time-out-as-expected-in-mulesoft-4.png) ## Testing the MuleSoft flow However, the flow behavior is not as expected. What happens is that the database connector waits for the execution of the stored procedure to finish (regardless of how long the execution takes) and then returns the response message established in the set payload. This indicates that the timeout configuration is not working properly. ![Postman GET returning 200 OK after 7.29 s, ignoring the 2-second timeout](../../assets/blog/oracle-database-stored-procedure-does-not-time-out-as-expected-in-mulesoft-5.png) The endpoint response takes more than 5 seconds. This is understandable (but not expected) since the stored procedure is waiting between 5 and 10 seconds before returning a response. ## Fixing the issue The solution to the problem is somewhat simple. Doing some research in the MuleSoft documentation, we can find that the origin of the problem is not our code but the Oracle driver: [https://help.mulesoft.com/s/article/Database-connector-Query-timeout-not-working-as-expected-on-oracle-database](https://help.mulesoft.com/s/article/Database-connector-Query-timeout-not-working-as-expected-on-oracle-database) The documentation indicates that if we put the parameter **-Doracle.net.disableOob=true** at runtime startup, this problem will be solved. To do this in a local environment, it is only necessary to go to the menu: Run > Run Configurations > Arguments and add the parameter in the startup configuration in Anypoint Studio and run the application again. ![Run Configurations Arguments tab adding -Doracle.net.disableOob=true to the VM arguments](../../assets/blog/oracle-database-stored-procedure-does-not-time-out-as-expected-in-mulesoft-6.png) In the following executions, we will always get an error indicating that the user (in this case the MuleSoft application) canceled the operation with the database. This is because the stored procedure is taking between 5 and 10 seconds to respond and the database connector is waiting a maximum of 2 seconds for the operation to complete. ![Postman now returning 500 Server Error with ORA-01013 user requested cancel of current operation](../../assets/blog/oracle-database-stored-procedure-does-not-time-out-as-expected-in-mulesoft-7.png) This means that the timeout parameter is now working properly. ## Conclusion As a conclusion, we can say that if we have the following mix: **MuleSoft + Oracle Database long-running executions (queries or stored procedure calls) + Timeout configuration** It is necessary to configure the Oracle driver parameter at Runtime startup to prevent the database connector from waiting for the execution to finish. Instead, the timeout configuration would be honored. **Additional notes**: You can notice that in the second POSTMAN call we have a response time of 6.59 seconds. That is also a problem on the Oracle JDBC driver side of things. The timeout configuration is honored, but the driver takes a long time to get the control back to the MuleSoft app. This was isolated and tested with a simple Java class and it has the same behavior. So, if you want to solve that problem, you’ll need to raise a ticket with the Oracle support team to solve the whole issue. --- ## Anypoint Platform Single Sign-on (SSO) SAML Configuration with Oracle IDCS – PART 1 Source: https://prostdev.com/post/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1 | Published: Feb 1, 2022 | Category: Tutorials It is very common to have the need to login into the Anypoint Platform console ([https://anypoint.mulesoft.com](https://anypoint.mulesoft.com/)) using what the customer already has in terms of an Identity Provider and an Identity Repository (LDAP). This could be for different reasons, such as: - Promote the current Single Sign-on (SSO) functionalities, provided by the customer’s Identity Provider. - Avoid the creation of new credentials and therefore, have the user memorize a new set of user/password to connect to Anypoint Platform. It is also common for customers to use OKTA, OpenAM, AWS Cognito, Microsoft Azure, etc. as their Identity Provider. And with it, it is normal to configure Anypoint Platform to connect with them using OpenID or SAML. I have another post (in Spanish) on how to make the connection with OKTA and Anypoint Platform. You can find it [here](https://mulesoftdevelopersmx.blogspot.com/2020/10/mulesoft-anypoint-platform-openid.html). On the internet, you will find information about how to configure SSO for Anypoint Platform, but with Identity Providers like OKTA or the ones I’ve mentioned at the beginning of this article. In this post, we will talk about how to do it using Oracle Identity Cloud Services (IDCS), which is another Identity Provider alternative, and it may be an interesting one, if you are using MuleSoft to connect to Oracle SaaS Applications, for example. ## Prerequisites The prerequisites for making this configuration are: - An active Anypoint Platform Instance. It can be a 30-day trial. - An active Oracle Identity Cloud Services tenant (can be a 300 credits free trial). - Some knowledge on SAML 2.0. - Understanding the role of an Identity Provider and a Service Provider. You can read [this article](https://www.okta.com/identity-101/why-your-company-needs-an-identity-provider/) to understand their differences. In our case, Anypoint Platform is acting as the Service Provider and Oracle IDCS is working as the Identity Provider (IdP). ## IDCS Configuration The first thing we need to do is to create an application in Oracle Identity Cloud Service that will generate the metadata that later we will import into Anypoint Platform. Let’s do it! Log in into your Oracle IDCS tenant and create a new application: ![Oracle IDCS Add Application dialog with SAML Application option highlighted](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-1.png) Click on SAML Application, the following screen will appear: ![App Details form with Name and Application URL / Relay State fields highlighted](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-2.png) Fill in the parameters: - Name: This can be anything and is the name that will represent your Application. In my case: MuleSoft Anypoint Platform - Application URL/Relay State: This needs to be[https://anypoint.mulesoft.com](http://anypoint.mulesoft.com/) Then click on the Next button: ![Step wizard at Details step with the Next button highlighted](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-3.png) At the top of the next screen you will see this: ![SSO Configuration tab with Download Identity Provider Metadata button highlighted](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-4.png) Click on Download Identity Provider Metadata. That will generate an XML file with all the information from the Identity Provider (IDCS), which we will use to import at Anypoint Platform. ## Anypoint Platform Configuration Leave the IDCS screen open and log in to[https://anypoint.mulesoft.com](https://anypoint.mulesoft.com/) in another tab of your browser. At the main menu, click on Access Management: ![Anypoint Platform menu with Access Management highlighted](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-5.png) Once there, head into Identity Management: ![Access Management sidebar with Identity Providers menu item highlighted](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-6.png) You will see that there is a default Identity Provider: ![Identity Providers list showing the default Anypoint User Credentials provider](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-7.png) Which is Anypoint itself. And there is where the users are being created and maintained. But the intention of this post is to add a new Identity Provider and connect it to Oracle IDCS using SAML. Just click on the Add Identity Provider blue button and select SAML: ![Add Identity Provider dropdown open with SAML 2.0 option highlighted](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-8.png) Once you are on the SAML 2.0 configuration page, it will allow you to import the metadata XML file that we’ve downloaded from the IDCS console in previous steps: ![New Identity Provider page with Import IdP Metadata file chooser highlighted](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-9.png) Click on the choose file, browse to the location where you’ve downloaded it, and import it. You will see that it will fill almost all the parameters on the screen: ![SAML config fields auto-filled from metadata: Sign On URL, Issuer, Public Key, Audience](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-10.png) The ones that are marked with orange were filled automatically, and the blue ones are the ones where you need to make some decisions: - Name: This is an arbitrary name that will identify this configuration. - Audience: This is also an arbitrary name, but in this case, this parameter is very relevant, since it will match with the configuration at the IDCS side. Then just click on Save Changes: ![Single Sign-On Initiation set to Both with the Save Changes button](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-11.png) You will have two Identity Providers configured: ![Identity Providers list now showing both Anypoint and Oracle IDCS](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-12.png) ## Get Important Information Before we get back to the Oracle IDCS console to finalize the configuration, we need to make a couple of things: - Obtain the audience that needs to match with the Entity ID at the IDCS side - Obtain the assertion consumer URL - Obtain the certificate key from Anypoint - Get the login URL that your Anypoint users will use to log in through Oracle IDCS To get those four things, simply click on the identity provider that you have just configured (in my case Oracle IDCS): 1. Get the audience that needs to match with the Entity ID at the IDCS side: ![Public Key field and highlighted Audience value rolps.anypoint.mulesoft.com](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-13.png) 2. Assertion consumer URL is taken from here: ![Configuration tab showing the Assertion Consumer Service (ACS) URL field](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-14.png) 3. The certificate key is taken from the Anypoint Keys tab. Once there, simply click on the download button that I am marking in yellow. This will download a .pem file: ![Anypoint keys tab with the primary key's download button highlighted in yellow](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-15.png) 4. The login URL can be copied from here: ![Identity Providers list with the external login domain URL highlighted](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-16.png) ## Putting It Together Now get back to Oracle IDCS console and fill the next parameters, accordingly to what we have just explained in the previous points: ![Oracle IDCS General SSO settings: Entity ID, Assertion Consumer URL, NameID, Signing Certificate](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-17.png) - In the Entity ID, copy the value of the audience that we’ve copied from the Anypoint Platform console - In the Assertion Consumer URL, copy the value that we’ve copied from the Anypoint Platform console - In the Signing Certificate, upload the .pem file that we’ve downloaded from the previous steps We are almost done, we just need to map the email attribute that will be returned from Oracle IDCS and that Anypoint Platform will map it: ![Attribute Configuration mapping an email attribute to the Primary Email user value](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-18.png) We just need to add **email** with the configuration we are showing in the image. After that, save your Oracle IDCS application and activate it. We are ready to test. Open a new browser and type the URL that we’ve copied from previous steps. It must be something like this: ``` https://anypoint.mulesoft.com/login/domain/ ``` You will see this: ![Anypoint login page with a Continue with Oracle IDCS button above username and password fields](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-19.png) You now have two options to login to Anypoint Platform and as you can see the name of your Identity Provider Configuration appears in the button. If you click there, you will be redirected to Oracle IDCS default login form: ![Oracle Cloud sign-in form in Spanish with username and password fields](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-20.png) Use your Oracle IDCS credentials, and you will get logged in: ![Anypoint Platform dashboard after login showing Design Center and DataGraph tiles](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-21.png) You will see this in the Users list at the Anypoint Platform: ![Anypoint Users list with the Identity Provider column showing Anypoint and Oracle IDCS](../../assets/blog/anypoint-platform-single-sign-on-sso-saml-configuration-with-oracle-idcs-part-1-22.png) (I am deleting sensitive data for security reasons). But you can see that it mapped the Username with the email provided by Oracle IDCS, and the email. You can also verify that the Identity Provider for that user is Oracle IDCS. But you can be questioning yourself, what happened with the name? Why was it not mapped? **Well, we will talk about that in the next article. We will elaborate on how to map attributes and roles coming from the Identity Provider, and mapped in Anypoint Platform.** --- ## How to check if a key is present in a JSON payload in DataWeave with these 3 examples Source: https://prostdev.com/post/how-to-check-if-a-key-is-present-in-a-json-payload-in-dataweave-with-these-3-examples | Published: Jan 18, 2022 | Category: Tutorials “In DataWeave, when a field in the input payload has a value and you append a **question mark (?)** at the end of the field, it will return **true**. On the other hand, if the field doesn’t exist, then it would return **false**.” Refer to the link below for more information: [https://blogs.mulesoft.com/dev-guides/how-to-tutorials/implement-logic-handling-in-dataweave/](https://blogs.mulesoft.com/dev-guides/how-to-tutorials/implement-logic-handling-in-dataweave/) Here are 3 examples where you can see how to check if a key in the input JSON payload is present or not. ## 1. Single Object In this example, I have used a single JSON object to find if a key is present or not and the result will be a boolean value. **Input:** ```json { "id":1, "company":"abc", "Address":"USA", "Phone":"123" } ``` **DataWeave Expression:** ```dataweave %dw 2.0 output application/json --- payload.id? ``` **Output:** ``` true ``` ## 2. Multiple Objects For this example, we have multiple objects in an array and we are going to check if the key is present or not with different DataWeave Expressions which will give us different results. In **DataWeave Expression 1** the output returned is just a boolean value which helps to combine the result. But we won't be able to know which object has **id** present in it. It just checks if there’s at least one value that matches the key and gives us a boolean value. In **DataWeave Expression 2** we will iterate through each item and then we can find out which item has **id** present in it. It gives complete information about a single object and the presence of the key in it. **Input:** ```json [ { "company":"abc", "Address":"USA", "Phone":"123" }, { "id":1, "company":"def", "Address":"UK", "Phone":"345" } ] ``` **DataWeave Expression 1:** ```dataweave %dw 2.0 output application/json --- payload.id? ``` **DataWeave Expression 2:** ```dataweave %dw 2.0 output application/json --- payload map ((item, index) -> item.id?) ``` **Output 1:** ``` true ``` **Output 2:** ```json [ false, true ] ``` ## 3. Nested Objects In this example, we have objects present in another object (nested objects). In this case, we will first go to the parent object and then to the key to find out whether it is present or not. **Input:** ```json { "id":1, "company":"abc", "Address":{ "street":"xyz", "zip":"080" }, "Phone":"123" } ``` **DataWeave Expression:** ```dataweave %dw 2.0 output application/json --- payload.Address.zip? ``` **Output:** ``` true ``` We have learned how we can check if the key is present or not within a different set of inputs and also observed different results obtained. I hope this was useful to you. --- ## MuleSoft Connected with Oracle Streaming Service using the Kafka Connector Source: https://prostdev.com/post/mulesoft-connected-with-oracle-streaming-service-using-the-kafka-connector | Published: Jan 4, 2022 | Category: Tutorials Apache Kafka is a very popular streaming service, and I could say that it is the leader in the market. Is part of the Apache Initiative, but vendors like Confluent offer a managed service that can be very powerful for organizations who are looking to start using the platform, without the need to control/manage the underlined infrastructure. But there are other options for streaming services that can be also useful for organizations who don’t want to get too much into the Kafka complexity. And in my case, since I am constantly working with Oracle Cloud Infrastructure, I’ve found a very interesting option for Kafka in Oracle Streaming Service. Oracle Streaming Service is a managed service running on top of Oracle Cloud. Is a serverless service, that users just need to provision and start creating streams/partitions that can be used for multiple purposes. With MuleSoft, it is common to have the need to connect to a Kafka Topic to consume a stream of messages. I’ve found that a very common scenario, for example in Retail, where they need to constantly send information from their Point-of-Sale to a central location. And if you think of a retailer who has multiple branches all over the country, then a stream makes a lot of sense. And for integration purposes, to have MuleSoft in the middle to enrich, transform and ultimately push the information to a system-of-records, then this mix of platforms offers you a very strong alternative. In this article we will show how to connect MuleSoft with Oracle Streaming Service, using the Kafka compatibility mode that this Oracle service offers. The first thing we need to do is to create a stream in Oracle Cloud: ![Oracle Cloud Streams list with the blue Create Stream button highlighted](../../assets/blog/mulesoft-connected-with-oracle-streaming-service-using-the-kafka-connector-1.png) That is a very straight-forward step, we just need to click on that blue button and fill in these parameters: ![Create Stream form with numbered name, compartment, stream pool, and partitions fields](../../assets/blog/mulesoft-connected-with-oracle-streaming-service-using-the-kafka-connector-2.png) - The name for your stream - Select the compartment where you want to create the stream - Create a New Stream Pool - Define the number of partitions And that’s it. You don’t need to worry about anything extra. Now, let’s move to MuleSoft. Imagine we need to create a MuleSoft application that consumes the stream that we have just created. MuleSoft does not have a connector to OCI Streams, but it does have a connector for KAFKA. Let’s look to a very simple application: ![Anypoint Studio Mule flow with a Kafka Message listener feeding a Logger](../../assets/blog/mulesoft-connected-with-oracle-streaming-service-using-the-kafka-connector-3.png) As you can see, we are using the KAFKA connector. We have the Message listener processor to listen into the stream and a simple Logger processor to print in the log file the payload/message. Now let’s see what the connector connection is going to ask us: ![Kafka Consumer config with the connection-type dropdown open on SASL/PLAIN Connection](../../assets/blog/mulesoft-connected-with-oracle-streaming-service-using-the-kafka-connector-4.png) The first thing to highlight is the Connection type. It offers different options: a) Consumer Plaintext Connection b) Consumer Kerberos Connection **c) Consumer SASL/PLAIN Connection** d) Consumer SASL/SCRAM Connection e) Consumer SASL/TOKEN Connection The one we need to use is **option c**. Also notice that we need to configure the bootstrap server, and as you can see, we’ve already filled in that information. We will show you where we get that info in the next paragraphs. After that, we need to define our **Group ID**. That is something you need to define, in my case: `test-consumer-group`. If we scroll down a little bit more, we will find the following parameters: ![Kafka Consumer config showing the lab-stream topic and SASL/PLAIN username and password](../../assets/blog/mulesoft-connected-with-oracle-streaming-service-using-the-kafka-connector-5.png) Which is: - Topics. In my case `lab-stream` - **SASL/PLAIN**. Which is the username that has access to the streaming service, and the password which is the AUTH Token that you will create in the next steps. Then, a particular step that can be tricky, is that we need to tell the connector settings that we want to use **SSL**, and for that, we need to go to this tab: ![Kafka Consumer Security tab with TLS Configuration set to Edit inline to enable SSL](../../assets/blog/mulesoft-connected-with-oracle-streaming-service-using-the-kafka-connector-6.png) And enable the TLS Configuration. Even though there will be no configuration at this tab, just please enable it. And that’s it, that is all we need. But the normal question will be: **but from where can I get that information from the OCI Streaming Service?** Well, let’s get back to the OCI Console and click on the Default Pool of your newly created stream: ![OCI Streams list with the stream's DefaultPool stream pool highlighted](../../assets/blog/mulesoft-connected-with-oracle-streaming-service-using-the-kafka-connector-7.jpg) Once there, at the left of the screen: ![DefaultPool stream pool page with the Kafka Connection Settings link highlighted](../../assets/blog/mulesoft-connected-with-oracle-streaming-service-using-the-kafka-connector-8.png) Click on the Kafka Connection Settings, It will show you this: ![Kafka Connection Settings showing bootstrap server, SASL connection string, and SASL_SSL](../../assets/blog/mulesoft-connected-with-oracle-streaming-service-using-the-kafka-connector-9.png) The bootstrap server is what we input in the connection window in MuleSoft. The username is the one that appears in the SASL Connection String box. Look to the element username, and copy the whole string, it will be something similar to this: ``` pedrito/oracleidentitycloudservice/pedrito@me.com/ocid1.streampool.oc1.iad.gz2gqeeegfljtyb2zr6cfxua ``` And for the **password**, you need to create an `AUTH_TOKEN`. For that regard, go to the user page: ![OCI user Auth Tokens page with the Generate Token button to create the password token](../../assets/blog/mulesoft-connected-with-oracle-streaming-service-using-the-kafka-connector-10.png) Click on the **Auth Tokens**, and generate a new one. Write it down immediately, since this is going to be the only time you will see it. Now we are ready to run our application in our Anypoint Studio and start consuming the stream. Let’s produce a message from the Streaming Service console: ![Test Stream panel producing a sample JSON glossary message to the lab-stream](../../assets/blog/mulesoft-connected-with-oracle-streaming-service-using-the-kafka-connector-11.png) And from the Console output in our Anypoint Studio we can see the message being consumed: ![Anypoint Studio console logging the JSON glossary message consumed from the stream](../../assets/blog/mulesoft-connected-with-oracle-streaming-service-using-the-kafka-connector-12.png) What do you think? This article is focused on highlighting MuleSoft’s KAFKA connector, to be used with OCI Streaming Services. --- ## CVE-2021-44228: Critical zero-day Log4j vulnerability (Log4Shell) discovered and MuleSoft’s solution Source: https://prostdev.com/post/cve-2021-44228-critical-zero-day-log4j-vulnerability-log4shell-discovered-and-mulesoft-s-solution | Published: Dec 15, 2021 | Category: News Over December 2021, a critical vulnerability has been unraveled pertinent to the popular Apache’s Log4j library. There is a high possibility that the code you have been working on this past few weeks has been using this component. So, let’s understand what is happening and what has happened. ## What is Apache Log4j? Apache Log4j is one of the most popular logging libraries and is a part of the Apache Logging Project. This has been one of the easiest and most effective libraries to enable logging for applications and is extensively used in Java applications. ## What is RCE (Remote Code Execution)? According to [Wikipedia](https://en.wikipedia.org/wiki/Arbitrary_code_execution): “arbitrary code execution (ACE) is an attacker's ability to run any commands or code of the attacker's choice on a target machine or in a target process. An arbitrary code execution vulnerability is a security flaw in software or hardware allowing arbitrary code execution. A program that is designed to exploit such a vulnerability is called an arbitrary code execution exploit. The ability to trigger arbitrary code execution over a network (especially via a wide-area network such as the Internet) is often referred to as remote code execution (RCE).” ## What happened? Recently, a critical zero-day vulnerability has been discovered which impacts this ever-popular logging library. All versions of Log4j2 versions from 2.0-beta9 to 2.14.1 have been affected by this vulnerability and currently, this is being exploited in the wild. This vulnerability enables an option for RCE (Remote Code Execution) on the vulnerable server and has been rated at a severity level of 10.0 by CVSS. This has been globally addressed as **CVE-2021-44228**. > [!NOTE] > To learn more about Log4j 2.x Security Vulnerabilities, visit their [official site](https://logging.apache.org/log4j/2.x/security.html). ## What made this vulnerability so dangerous? [CVE-2021-44228](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228), also known as Log4Shell, is a vulnerability that enables RCE. The fact that it takes only a few lines of code to activate this makes this so much more dangerous. The attacker only needs to force into one of the servers to potentially gain access to the entire system which can be done by simply adding just one string to the log. Post this an entire arbitrary code can be uploaded into the application. ## Why did this happen? This potentially happened due to a loophole in the JNDI (Java Naming & Directory Interface) which is an API that enables a Java application to perform lookups on any type of named Java objects. Currently, in this case, the JNDI has not been restricted against unknown look-ups. An attacker can send an artificially constructed request to an application employing a vulnerable version of Log4j. This request then can enforce an LDAP / LDAPS injection by using JNDI (which is not validating the authenticity of the incoming request) which then sends a malicious / ill-intended payload to the application from one of the attacker-controlled servers. This payload can contain a piece of arbitrary code (or a Java class) that can enable RCE. ## What can be done to fix this? There are a couple of ways which have been discovered to fix this: - One can update the Log4j package to version `2.17.0` which has been recently released by Apache. (**RECOMMENDED**) - In the case of Log4j versions from 2.10 to 2.14.1, one can update the system property `log4j2.formatMsgNoLookups` or the environment variable `LOG4J_FORMAT_MSG_NO_LOOKUPS` to `true`. - For Log4j versions before 2.10, one can simply remove the `JndiLookup` class from the classpath: `zip -q -d log4j-core – *. Jar org / apache / logging / log4j / core / lookup / JndiLookup .class`. ## How to fix it in MuleSoft **CloudHub** customers: If you are running on CloudHub, MuleSoft has released the fix for this vulnerability. The fix is scheduled to be applied to all runtimes in CloudHub starting Dec 11th, 2021 9 PM PDT to Dec 14th, 2021 10 PM PDT. **On-Premise** customers: This issue requires immediate attention. MuleSoft strongly recommends all on-premise customers take action to update their Mule runtimes as soon as possible. Mule runtime engines associated with the following products will need to be patched as well: - Anypoint Studio - Runtime Fabric (RTF) - Pivotal Cloud Foundry (PCF) - Private Cloud Edition (PCE) For details about how to fix this issue in Mule 4 or Mule 3, please follow the instructions on this MuleSoft article: [https://help.mulesoft.com/s/article/Apache-Log4j2-vulnerability-December-2021](https://help.mulesoft.com/s/article/Apache-Log4j2-vulnerability-December-2021) ## How to update log4j from Maven Go to `mvnrepository.com` and search for `log4j-core` or click on [this link](https://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-core). This will show you the latest version available at the top and if there are any vulnerabilities discovered for any specific version. ![mvnrepository.com log4j-core page listing versions with vulnerability counts per release](../../assets/blog/cve-2021-44228-critical-zero-day-log4j-vulnerability-log4shell-discovered-and-mulesoft-s-solution-1.png) Click on version `2.17.1` to get the code for your `pom.xml` file. This will open a new view: ![log4j-core 2.17.1 detail page showing the Maven dependency XML snippet to copy](../../assets/blog/cve-2021-44228-critical-zero-day-log4j-vulnerability-log4shell-discovered-and-mulesoft-s-solution-2.png) Simply click on the XML text under **Maven** and this will copy the code to your clipboard. Go to any of your Mule projects and open the `pom.xml` file located at the root folder of your project. Paste the copied code into the `` section. ![pom.xml dependencies section with the log4j-core 2.17.1 dependency block highlighted](../../assets/blog/cve-2021-44228-critical-zero-day-log4j-vulnerability-log4shell-discovered-and-mulesoft-s-solution-3.png) That’s it! This will update your log4j version in the `.m2` folder. To verify, you can go to your `.m2` folder and follow this path: `.m2/repository/org/apache/logging/log4j/log4j-core` - there should now be a folder called `2.17.1`. ![Terminal ls -la of the .m2 log4j-core folder listing version subfolders including 2.17.1](../../assets/blog/cve-2021-44228-critical-zero-day-log4j-vulnerability-log4shell-discovered-and-mulesoft-s-solution-4.png) After that, remove the previous versions to make sure you only have the latest version. You can remove other folders using `rm -rf ... `. For example, `rm -rf 2.13.3/ 2.14.0/`. ## Updates since this post was published - Apache Log4j version 2.16.0 is now available - Apache Log4j version 2.17.0 is now available - Added section "How to update log4j from Maven" We hope this information has been helpful in knowing what has happened and what can be done to mitigate the risk. As this issue is still being investigated, we will keep posting with any latest developments on this. Also, you can comment if you want to add anything to this article / if anything needs to be changed. Thank you!! - [Soumyajit Sinha](https://www.prostdev.com/profile/soumyajitsinha888/profile) - [Leonardo Gonzalez](https://www.prostdev.com/profile/32856a2a-f548-4d76-a415-447e4b1aa2d4/profile) - [Alex Martinez](https://www.prostdev.com/profile/alexandramartinez/profile) --- ## Anypoint Studio 7.11.0: New features overview Source: https://prostdev.com/post/anypoint-studio-7-11-0-new-features-overview | Published: Dec 14, 2021 | Category: News Are you already using the newest version of Anypoint Studio? If the answer is yes, let me guide you through the coolest features it has. If the answer is no, go ahead and download it [here](https://www.mulesoft.com/lp/dl/studio) to start playing with it! ## New features Anypoint Studio includes three new features that you will enjoy as a MuleSoft Developer: - Support for Mule Runtime 4.4. - Correlation ID Management. - Improved Logging Capabilities. - DataWeave Updates (DataWeave 2.4 features!). - Mule Tracing Module. - Feature Flags. - DataWeave 2.4. Including language improvements and new modules. - Read larger-than-memory strings automatically. - New modules, functions, types, annotations, and variables. - Helper functions for handling null values. - “Referenced by” function. Now you can see a list of all flow references to a particular flow/subflow and you also can jump there with a few clicks. If you are interested in learning more about Mule 4.4 new features, you will find all the details [here](https://docs.mulesoft.com/mule-runtime/4.4/whats-new-in-mule). Also, if you are an amazing DataWeave programmer, it’s a good idea to take a look at the [docs](https://docs.mulesoft.com/dataweave/2.4/whats-new-in-dw). As soon as you start your Anypoint Studio installation, you will see the following: ![Anypoint Studio 7.11.0 welcome screen highlighting Mule 4.4 and DataWeave 2.4 as new features](../../assets/blog/anypoint-studio-7-11-0-new-features-overview-1.png) ## Support for Mule 4.4 In this new version of Studio, you will have available the newest version of the Mule Runtime (4.4). When you create a new project just select Mule Server 4.4.0 EE and start enjoying the new features: ![New Mule Project dialog with the Runtime list showing Mule Server 4.4.0 EE selected](../../assets/blog/anypoint-studio-7-11-0-new-features-overview-2.png) As you can notice you are using the Mule 4.4 version. You will find it everywhere in your project (`pom.xml` file, dependencies, and so on). ![Anypoint Studio project with the pom.xml showing the app.runtime property set to 4.4.0](../../assets/blog/anypoint-studio-7-11-0-new-features-overview-3.png) You will also find this version within the `mule-artifact.json` file: ![mule-artifact.json file with minMuleVersion set to "4.4.0"](../../assets/blog/anypoint-studio-7-11-0-new-features-overview-4.png) The new Tracing Module is already in the Mule Palette: ![Mule Palette search for "tracing" listing the Tracing module's logging-variable operations](../../assets/blog/anypoint-studio-7-11-0-new-features-overview-5.png) ## DataWeave 2.4 There are various new DataWeave features, modules, and functions. Let’s try some of them. ### onNull function The onNull function executes a callback function if the preceding expression returns a null value and then replaces the null value with the result of the callback. In the following DataWeave script, we are concatenating ‘Hello’ with the content of the payload. But, what happens if the payload has a null value? We’ll receive an error like this: ![DataWeave error from concatenating "Hello " with a null payload, with the warnings dialog open](../../assets/blog/anypoint-studio-7-11-0-new-features-overview-6.png) If we fix the input payload value, then the DataWeave script will return the expected (concatenated) value: ![DataWeave concatenation returning "Hello World!" once the payload is no longer null](../../assets/blog/anypoint-studio-7-11-0-new-features-overview-7.png) How can we avoid this type of error using DataWeave and particularly the onNull function? ![DataWeave using onNull to replace a null payload, returning "Hello, this is a null payload."](../../assets/blog/anypoint-studio-7-11-0-new-features-overview-8.png) ### hammingDistance function Let’s try another new function that caught my attention: hammingDistance. According to Wikipedia, in information theory, the Hamming distance between two strings of equal length is the number of positions at which the corresponding symbols are different. Let me explain this with an example: We have two strings: - String 1 = 11011001 - String 2 = 10011101 Both strings have eight characters of length and they are different only at positions 2 and 5. So they have two positions at which their characters are different, then the result of the hamming distance should be 2. We can confirm this using the DataWeave function. Notice that the hamming distance function is within the Strings module, so it must be imported explicitly in your DataWeave script. ![DataWeave hammingDistance comparing two 8-bit strings and returning 2 for the two differing positions](../../assets/blog/anypoint-studio-7-11-0-new-features-overview-9.png) Those are only two of the new functions that were included in DataWeave 2.4. Please explore the docs and enjoy all the new features, modules, and functions. ## “Referenced by” function Finally, let’s review this new feature called “referenced by”. This could be useful when you have multiple configuration files within a complex Mule application and you want to jump to the flows that are referencing a particular flow or subflow. The only thing you need to do is right-click on the flow (or subflow) you are currently working on and then select the “Referenced by” option (Ctrl+Shift+G). ![Right-click flow menu in Anypoint Studio with the "Referenced by" option highlighted](../../assets/blog/anypoint-studio-7-11-0-new-features-overview-10.png) The result will be a list of flows that are referencing that particular flow and then you can jump to any of them. ![Mule Configuration tree listing the two flow references to the sub-flow as Ref #1 and Ref #2](../../assets/blog/anypoint-studio-7-11-0-new-features-overview-11.png) When you double-click over the reference, automatically you will be positioned on that particular flow reference: ![Double-clicking a reference jumps the canvas to the sub-flow's Flow Reference in parent-flow](../../assets/blog/anypoint-studio-7-11-0-new-features-overview-12.png) It is cool, isn't it? Now you have explored the newest version of Anypoint Studio, please let us know what do you think? What are the new things you found on it? And… happy coding! --- ## Retry and Reprocess flow design with Anypoint MQ Source: https://prostdev.com/post/retry-and-reprocess-flow-design-with-anypoint-mq | Published: Nov 16, 2021 | Category: Tutorials *GitHub repository with the Mule project and the spring boot application can be found at the end of the post.* In any MuleSoft project, when Anypoint MQ comes into the picture there is always a requirement of retry and reprocess mechanism. Based on my experience I have prepared this blog to show how easily we can design retry and reprocess mechanisms. ## Requirement The main flow of the Mule app will subscribe to a queue and the payload has to be sent to the backend via an HTTP call. While sending the subscribed payload to the backend via HTTP. If the backed system is down (during sending the HTTP request), how are we going to define our flow to perform the retry and reprocess mechanism? ## Step 1 Put the HTTP request inside an **until-successful scope**. Which is a default scope provided by MuleSoft for retrying any event. Define the below parameters in the scope: - **maxRetries**: Number of retries the request will do to try to connect with the backend service. - **millisBetweenRetries**: After the retry performs, how much time the application should hold for the next retry. Code for backend call using until-successful: ```xml ``` ## Step 2 If we can't connect to the backend even after using the until-successful scope (retry process), we have to introduce a reprocess mechanism. To enable the reprocess mechanism, we have to call a subflow (which will perform reprocess) from the "**On Error Continue**" of the main flow. > [!IMPORTANT] > We have to acknowledge the subscribed queue in both HAPPY or ERROR scenarios. If we don't do that, the queue will hold the message and it may lead to a duplicate process. ![Mule flow with HTTP Request in a Retry to connect scope and an On Error Continue reprocess subflow](../../assets/blog/retry-and-reprocess-flow-design-with-anypoint-mq-1.png) ## Step 3 To introduce the reprocess mechanism we have to define **two** queues. **1. Dead letter queue** This queue will work as an intermediate queue. This means, after failing the retry process we will store the message in this queue for re-processing. While sending the message to DLQ, we are going to pass two values in MQ properties: - **reprocessCount**: This value is going to be used as a counter of the reprocess call. - **qname**: The main queue name from which our application is subscribing to the messages. Apart from the above two MQ properties, we have to define a property in the properties file. Which will be the maximum re-process count. - **maxUserRegistrationReprocessAttempts: "3"** - This means the reprocess will happen (Main Queue - DLQ) only three times. **2. Parking queue** This queue will be treated as the final destination of the processing message. This means if both mechanisms (retry and reprocess) fail, the message will be stored in this queue (which will reprocess again). The reason for introducing the parking queue is 100% delivery/processing of each message (at any condition). We can introduce an additional flow, which will reprocess the landed message from parking to the main queue. We can define this in many ways. One of the ways is: Once the parking queue is filled with a certain limit an email will trigger to the support team. After receiving the email, the support team can trigger an endpoint that will start pulling the messages from the parking queue and pushing them to the main queue (using parallel for each component). ## How to test the POC Deploy the Mule app in Anypoint Studio and the spring boot app in eclipse or CMD. Define the main queue, dead letter queue, and parking queue according to your requirements and update the CI/CS of the Anypoint client app (by using the MQ portal). ![Properties file AnypointMQ config with client_id, queue, deadLetter, and parking queue settings](../../assets/blog/retry-and-reprocess-flow-design-with-anypoint-mq-2.png) **Happy flow**: After the successful deployment of both Mule and the backend application, send a message as shown below. The flow will execute without any error. ![Anypoint MQ console sending a JSON user-registration message to the queue](../../assets/blog/retry-and-reprocess-flow-design-with-anypoint-mq-3.png) **Error flow**: To test the retry and reprocess flows, we have made the backend unavailable. So, we have to deploy only the Mule app (should not deploy the spring boot app) and send the message like above and observe the process in the log. After performing the retry and reprocess flows, the message will land in the parking queue as below. ![Parking queue showing a JSON message with reprocessData that landed after retries failed](../../assets/blog/retry-and-reprocess-flow-design-with-anypoint-mq-4.png) ## Conclusion This POC is just to give an idea about retry and reprocess design using Anypoint MQ. There will be a lot of scopes to enhance the POC and prepare as per your requirement. Finally, I will be more than happy to receive your valuable feedback. ## GitHub repository Kindly find the complete Mule and backed (spring boot) application [here](https://github.com/anky123/ankuran.online.mule4/tree/master/project07%20%7C%20retry-process-with-mq). --- ## How to configure Solace DMQ (Dead Message Queue) through the Mule JMS connector Source: https://prostdev.com/post/how-to-configure-solace-dmq-dead-message-queue-through-the-mule-jms-connector | Published: Nov 9, 2021 | Category: Tutorials In this post, I will be explaining how to configure Solace DMQ through the Mule JMS Connector. There can be many business scenarios where the processing of the message will be delayed by a predefined amount of time. In such scenarios, we can make use of the concept of the TTL property and the DMQ. ## Dead Message Queues By default, Guaranteed messages are removed from a durable endpoint's message pool and discarded when: - The number of redelivery attempts for a message exceeds the Max Redelivery value for the original destination endpoint; - Or, a message's Time-To-Live (TTL) value has been exceeded and the endpoint is configured to respect message TTL expiry times. If you don't want the messages to be discarded, they can be moved to a dead message queue (DMQ) assigned to the endpoint. ## How DMQs Work Any durable queue on the same Message VPN as the endpoint that the messages are being removed from can be assigned as a DMQ. Messages flagged as DMQ-eligible by the publishing client will be sent to the endpoint's DMQ rather than be discarded. However, if an endpoint's assigned DMQ doesn't exist, then the removed Guaranteed messages will still be discarded even if they are DMQ-eligible. By default, each durable endpoint is assigned a DMQ named `DEAD_MSG_QUEUE`, but this queue does not pre-exist and must be created by a management user. Now let us see the steps involved in setting up DMQ and TTL. ## 1. Enabling the "Publisher Messages DMQ Eligible" property in the JMS JNDI of the Message VPN Log in to Solace Cloud Message VPN. (Please refer to my first blog, “[How to integrate Solace PubSub+ Cloud with MuleSoft](https://www.prostdev.com/post/how-to-integrate-solace-pubsub-cloud-with-mulesoft),” which explains how to create a Solace Cloud Trial account and how to spin up a Service in Solace). Click on JMS JNDI --> Click on Connection Factories --> Click on /jms/cf/default --> Scroll down and enable the property “Publisher Messages DMQ Eligible” as highlighted in the screenshot below. ![Solace JMS connection factory settings with "Publisher Messages DMQ Eligible" toggled on](../../assets/blog/how-to-configure-solace-dmq-dead-message-queue-through-the-mule-jms-connector-1.png) ## 2. Creating Queues and Topics Create the new Queues and their corresponding Topics as shown below. As part of the POC, our queues of interest are highlighted below. ![Solace Queues list highlighting test-queue-1 and test-queue-2](../../assets/blog/how-to-configure-solace-dmq-dead-message-queue-through-the-mule-jms-connector-2.png) ## 3. Setting up DMQ and TTL in the “test-queue-1” queue In the advanced settings of the `test-queue-1` enable the ‘**Respect TTL’** and set the ‘**Maximum TTL**’ as 180 seconds (this can be set based on our need). Also, set the ‘**DMQ Name**’ as `test-queue-2`. As per this setting, whenever any message reaches `test-queue-1`, the message stays in this queue for 180 seconds before it’s pushed to DMQ (`test-queue-2`). ![test-queue-1 settings: DMQ Name test-queue-2, Respect TTL on, Maximum TTL 180 seconds](../../assets/blog/how-to-configure-solace-dmq-dead-message-queue-through-the-mule-jms-connector-3.png) ## 4. Publishing Message to Solace Topic with SOLACE_JMS_PROP_DEAD_MSG_QUEUE_ELIGIBLE set to true In the `solace-jms-publish.xml`, I added an HTTP Listener to be able to listen to a request with the content that I want to publish into the Solace Topic. Here you can see how I have configured the JMS Publish connector: ![Anypoint Studio publish flow with HTTP Listener and JMS Publish to a Solace topic](../../assets/blog/how-to-configure-solace-dmq-dead-message-queue-through-the-mule-jms-connector-4.png) Also, I have set the property `SOLACE_JMS_PROP_DEAD_MSG_QUEUE_ELIGIBLE` to `true` in the **JMS Config Provider Properties.** ![JMS Config provider properties listing SOLACE_JMS_PROP_DEAD_MSG_QUEUE_ELIGIBLE](../../assets/blog/how-to-configure-solace-dmq-dead-message-queue-through-the-mule-jms-connector-5.png) ![dev.yaml properties file with the Solace JMS config and dmqEligible set to "true"](../../assets/blog/how-to-configure-solace-dmq-dead-message-queue-through-the-mule-jms-connector-6.png) ## 5. Publishing the message from Postman ![Postman POST to localhost:8081/publish returning 200 OK "published the message to Solace Topic"](../../assets/blog/how-to-configure-solace-dmq-dead-message-queue-through-the-mule-jms-connector-7.png) Now the message is published to the Topic with ‘**DMQ Eligible**’ set to ‘**Yes’**. This message stays in the Queue `test-queue-1` for 180 seconds before which it will be published to DMQ (`test-queue-2`) automatically for further processing. ![test-queue-1 Messages Queued tab showing one message with DMQ Eligible Yes](../../assets/blog/how-to-configure-solace-dmq-dead-message-queue-through-the-mule-jms-connector-8.png) ![test-queue-2 (the DMQ) Messages Queued tab showing the message after the TTL elapsed](../../assets/blog/how-to-configure-solace-dmq-dead-message-queue-through-the-mule-jms-connector-9.png) ## Recap Now we know how a delay in the processing of the messages can be achieved in Solace using the DMQ and TTL feature. Thanks for reading my post and I hope it will help! That’s it for now! See you in the next post! -Pravallika --- ## How to Handle Single and XA (Extended Architecture) Transactions in MuleSoft Source: https://prostdev.com/post/how-to-handle-single-and-xa-extended-architecture-transactions-in-mulesoft | Published: Oct 12, 2021 | Category: Guides "Transaction" is a word that is used way too frequently in our daily lives. We use it whenever we are engaged in purchasing or selling any commodity. In our daily lives, a transaction signifies an instance of buying or selling something. Also, a transaction can be deemed as completed only upon the overall completion of the buying/selling activity along with the monetary affair associated with it. In the software industry, we use the word ‘transaction’ in multiple areas. Predominantly, it is used while discussing SQL queries. Transactions make up a very important component of any software development life cycle (SDLC). Let’s see how transactions are changing lives! A transaction is a group of operations where all the steps are needed to be successful to commit a result. If any one of the intermediate steps fail, the whole chain of steps fails collectively. Let’s understand it using a pictorial reference. ![Two transaction flowcharts: one committing all four steps, one rolling back after an error at step 2](../../assets/blog/how-to-handle-single-and-xa-extended-architecture-transactions-in-mulesoft-2.png) Let’s consider a case where we have a chain of steps that represents the following: | Step | Action | Table involved | |---|---|---| | Step 1 | This is a SQL fetch operation that fetches information on a certain deal | Table A | | Step 2 | This is a SQL insert operation that inserts the new record after consolidating the data received in the previous step | Table A | | Step 3 | This is a SQL delete operation that deletes the previously available record | Table A | | Step 4 | This is a SQL update operation that updates the amount update in another table | Table B | We want all the steps to be completed collectively or else we want none of them to happen. In layman’s terms, we want the event to be committed only if all the steps are successful. If any intermediate step fails, we do not want the event to be committed and we do not want any step to be actioned upon. In the above picture, we have encapsulated these 4 steps in a **transaction**. The first diagram starts and completes without any errors and gets committed. But in the second diagram, we can see that in step 2 we encountered a certain error. Now, we don’t want the event to be committed. This is made possible by the **transaction** action. It will make sure that all the steps are rolled back to the initial state from which the event was kicked off. Now that we are clear on what transaction is and how it helps in software development, let’s indulge ourselves in bringing it to life in MuleSoft. MuleSoft supports two types of transactions: - Single resource - XA transactions (Extended Architecture) Let's talk about them in detail. ## Single Resource transactions Let's consider a Mule flow where we are dealing with a single mode of data transaction. Consider our flow is having only JDBC affairs or JMS activities or VM Queues. We have created a flow in which we are dealing with 2 tables connecting to the same database. This is how the flow looks like: ![single-resource-example flow in a Try scope: numbered Select, Insert, Delete, and Update database operations](../../assets/blog/how-to-handle-single-and-xa-extended-architecture-transactions-in-mulesoft-3.png) Here, the following database actions are carried out: | Step | Action | Table involved | |---|---|---| | Step 1 | A JDBC select operation (**1**) fetches information on a certain publication | Table A | | Step 2 | A JDBC insert operation (**2**) inserts the new record after consolidating the data received from the user & the data received in the previous fetch operation | Table A | | Step 3 | A JDBC delete operation (**3**) deletes the previously available record in the table in which data is inserted in step 2 | Table A | | Step 4 | A JDBC update operation (**4**) updates the author details of the publication in another table in the same database | Table B | Here, we have encapsulated all the JDBC components within a **TRY** block. **Why use the TRY block?** Try block is helping us to enforce the transaction on the JDBC operations by acting as the initiator of a transaction thread. The JDBC components simply connect themselves to the transaction initiated by the try component. In a try block, you can set how you want to deal with transaction action. ![Try scope settings with the Transactional action dropdown open showing ALWAYS_BEGIN, BEGIN_OR_JOIN, INDIFFERENT](../../assets/blog/how-to-handle-single-and-xa-extended-architecture-transactions-in-mulesoft-4.png) The try-block provides us the following transaction actions: | Transactional action | Description | |---|---| | `ALWAYS_BEGIN` | It will always initiate a new transaction | | `BEGIN_OR_JOIN` | It will initiate a new transaction if there are no preceding transactions. In case there is a transaction that has already been set up, the try block will join that transaction | | `INDIFFERENT` | It will be neutral. It will neither create a transaction nor join an existing transaction | *For more information on the Transactional Actions, please refer to the* [official documentation](https://docs.mulesoft.com/mule-runtime/4.3/transaction-management#transactional-actions)*.* Once you have defined the transaction action, you can go ahead and set the transaction type. ![Try scope settings with the Transaction type dropdown open showing LOCAL and XA options](../../assets/blog/how-to-handle-single-and-xa-extended-architecture-transactions-in-mulesoft-5.png) The try-block provides us the following transaction actions: | Transaction type | Description | |---|---| | LOCAL | Single Resource | | XA | Extended Architecture Transactions | In our case, since we are dealing with Single Resource (only JDBC operation), we will be selecting the LOCAL transaction type and we will let the try-block initiate the transaction. Once the transaction has been kicked off by the try block, all we need to do is to instruct our JDBC components to JOIN this transaction. We can do that simply by going to the “advanced” section of the JDBC operator. ![Database operation Advanced tab with the Transactional action dropdown set to ALWAYS_JOIN](../../assets/blog/how-to-handle-single-and-xa-extended-architecture-transactions-in-mulesoft-6.png) We will select `ALWAYS_JOIN` as the transactional action so that it joins the transaction initiated by the try block earlier. We will be selecting the same option for all our JDBC components. Once this is done, we can go ahead and deploy our code. ## Extended Architecture transactions Let’s consider a case when we need to deal with multiple data exchange sources. For example, if we need to use a JMS broker along with a JDBC operation and VM queues and we need all of these operations to be performed atomically. We want all these operations to be performed following the ACID (Atomicity, Consistency, Isolation & Durability) concept. | ACID property | Description | |---|---| | Atomicity | The entire transaction should be performed at a single go with all the components being successful | | Consistency | The results from all the operations should be consistent across all the sources/data exchanges | | Isolation | The operations should be performed separately and sequentially. There should be no overlap | | Durability | Once the transaction is committed, then under no circumstance it should be rolled back | The XA transaction follows a **2-Phase-Commit protocol**. The first phase of the 2PC is used to check whether all the actions/steps (we can consider these steps to be our JDBC/JMS components) in a transaction have been successfully completed. If all of them pass or get completed, the transaction manager commits the transaction and writes the corresponding completion logs which can be found in a file with a name ending with “tx-logs” in the .mule folder of your workspace directory. If any of the components fail execution, the transaction manager rolls back the entire transaction. We will be using **Bitronix Transaction Manager** as our transaction manager to accomplish this. We can add this in our Global Elements section in Anypoint Studio. ![Two dialogs: choosing the Bitronix Transaction Manager global type, then its properties with OK highlighted](../../assets/blog/how-to-handle-single-and-xa-extended-architecture-transactions-in-mulesoft-7.png) To add **Bitronix Transaction Manager**, head on to the Global Elements section and search for Bitronix (left image). Once done, another window (right image) will pop up. Click on “OK” and you are all set! Let’s try this out in Anypoint Studio. This is how the flow will look like in this case: ![extended-architecture-example flow adding a fifth step to publish to a JMS topic across two databases](../../assets/blog/how-to-handle-single-and-xa-extended-architecture-transactions-in-mulesoft-8.png) Here, the following database actions are carried out: | Step | Action | |---|---| | Step 1 | A JDBC select operation (**1**) fetches information on a certain publication from a table in **Database A** | | Step 2 | A JDBC insert operation (**2**) inserts the new record after consolidating the data received from the user & the data received in the previous fetch operation in a table in **Database A** | | Step 3 | A JDBC delete operation (**3**) deletes the previously available record in the table in which data is inserted in step 2 from a table in **Database A** | | Step 4 | A JDBC update operation (**4**) updates the author details of the publication in another table in the **Database B** | | Step 5 | Publish the author update details in a JMS Queue (**5**) | Compared to what we have done in our implementation of Single Resource transaction, we will do the same thing with the try block, JDBC & JMS components. The only difference is that we have to intimate Mule that we will be using XA transactions. To do that, we have to select XA as “transaction type” in the Try scope settings. For JDBC & JMS components, this is configurable in their Global Settings. In the Database connection config palette, we can head to the “transactions” section where we will have a tick-box to turn the XA transaction on. ![Database Config Transactions tab with the Use XA Transactions checkbox ticked](../../assets/blog/how-to-handle-single-and-xa-extended-architecture-transactions-in-mulesoft-9.png) In the JMS connection config palette, we need to scroll down to the “Connection Factor” section. Select “Edit inline” for the “Factory configuration”. Then we will have a tick-box to turn the XA transaction on. ![JMS Config with No caching strategy and the Enable xa checkbox ticked in the connection factory](../../assets/blog/how-to-handle-single-and-xa-extended-architecture-transactions-in-mulesoft-10.png) > [!NOTE] > For JMS Connection, XA will not work when caching is activated. Please select “No caching” as the Caching Strategy and then click on “Test Connection…” to see whether the connection is successful or not. We will select `ALWAYS_JOIN` as the transactional action so that it joins the transaction initiated by the try block earlier. We will be selecting the same option for all our JDBC and JMS components. Once this is done, we can go ahead and deploy our code. So, now we have the knowledge of what a transaction is and how to set up one in MuleSoft. Apart from that, we have talked about the various kinds of transactions and how to consider one based on the requirement. Following this up will be an article in which we will take a deeper dive into other intricacies of transactions. Thank you!! --- ## How to create a README file for your GitHub profile, part 2: Markdown basics Source: https://prostdev.com/post/how-to-create-a-readme-file-for-your-github-profile-part-2-markdown-basics | Published: Sep 28, 2021 | Category: Tutorials *GitHub repository with the final `README.md` file can be found at the end of the post.* Having a [GitHub](https://github.com/) account in today’s world is a **must-have** for any software developer. In this series of posts, I’ll show you how to create a striking `README.md` file to attach to your GitHub profile. If you want to see an example of the final result, take a look at [my GitHub profile](https://github.com/alexandramartinez)! If you’re new to Git, GitHub, or `README.md` files, make sure to check out the previous post of this series: [How to create a README file for your GitHub profile, part 1: Intro to Git, GitHub, and README files](https://www.prostdev.com/post/how-to-create-a-readme-file-for-your-github-profile-part-1-intro-to-git-github-and-readme-files). Now, I’ll guide you through the basics of creating a Markdown file. ### What is Markdown? [Wikipedia](https://en.wikipedia.org/wiki/Markdown) states that “Markdown is a lightweight markup language for creating formatted text using a plain-text editor. ... Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files.” In summary, it’s a language used to write files using formatted text (like bold or italic characters, bulleted lists, numbered lists, headers, links, code, etc.). Markdown files end in `.md` (like `README.md`). I like to look at it as simplified [HTML](https://en.wikipedia.org/wiki/HTML). But enough theory. You’ll learn more with practice. ### Create the repository to keep the README If you don’t already have a GitHub account, please create one first. Once you have it, click on your profile picture at the top-right of the page and click on “Your repositories.” ![GitHub profile dropdown menu with Your repositories highlighted](../../assets/blog/how-to-create-a-readme-file-for-your-github-profile-part-2-markdown-basics-1.png) Now click on the green “New” button. ![GitHub Repositories tab toolbar with the green New repository button](../../assets/blog/how-to-create-a-readme-file-for-your-github-profile-part-2-markdown-basics-2.png) **This is important** - you have to name your new repo exactly the same as your username. If you’re not sure what yours is, just click on your profile picture again, and you will see it after the “Signed in as …”. In my case, it’s “alexandramartinez”. You don’t have to add a description. Make sure to select “Public” and the “Add a README file” checkbox like in the screenshot below. ![New repository form set to Public with Add a README file checked, and Create repository button](../../assets/blog/how-to-create-a-readme-file-for-your-github-profile-part-2-markdown-basics-3.png) Click on “Create repository.” You now have a new repo with an (almost) empty README file. Click on the “edit” button on the right side of the `README.md` file. ![README.md preview showing the alexandramartinez heading with an edit pencil icon](../../assets/blog/how-to-create-a-readme-file-for-your-github-profile-part-2-markdown-basics-4.png) This will bring you to the editor view. You also have a Preview tab in order to see the formatting of the file. ![GitHub README editor with Edit file and Preview tabs showing the # alexandramartinez heading](../../assets/blog/how-to-create-a-readme-file-for-your-github-profile-part-2-markdown-basics-5.png) ### How to add headers in Markdown Ok, from this point on, I’ll mainly show you the code and the explanations of the formatting. I trust you’ll use the “Edit file” and “Preview” tabs to make sure it looks as expected. Replace your username with your own title. In my case, I wrote: ``` ## Hi, I'm Alex! ``` The hashtag character is used to specify headers. The single hashtag is the biggest one. You can use ##, ###, ####, and so on to specify what type of header you want to format (each smaller than the last one). In my README file, I’m just using the first title and the following header (H1 and H2). This is how my file looks so far, with all my headers and a brief introduction: ``` ## Hi, I'm Alex! I'm a Software Engineer who started creating content and is now a Developer Advocate! ### About me ### Latest Content ### GitHub Stats ### Connect ### MuleSoft Repos ``` ### How to add bulleted lists in Markdown Now, in the “About me” header, I am using a bulleted list to show my current positions. To create this list, you just have to use a dash (-) and leave a space between the dash and the following character. Like this: ``` - Developer Advocate at MuleSoft - Founder & Content Creator at ProstDev - Lead at Women Who Code Monterrey ``` ### How to add hyperlinks in Markdown I also want to add links to the companies I mentioned in this list. The syntax to add links in Markdown is the following: ``` [Text to display](link) ``` Here is what it looks like with the 3 links added to MuleSoft, ProstDev, and Women Who Code Monterrey: ``` - Developer Advocate at [MuleSoft](https://www.mulesoft.com/) - Founder & Content Creator at [ProstDev](https://www.prostdev.com/) - Lead at [Women Who Code Monterrey](https://www.womenwhocode.com/monterrey) ``` Finally, feel free to make your introduction a bit more fun or show who you are. In my case, I thought adding emojis was a bit more casual. Yes, you can also add emojis to Markdown! ### Final result (for now) Here’s the result we have so far (following my own README file): ``` ## Hi, I'm Alex! I'm a Software Engineer who started creating content and is now a Developer Advocate! - 🇲🇽 in 🇨🇦 - 🐱 🐶 Eris & Waffle - ❤️ Horror, Disney, Jurassic Park, Coffee, Halloween - 🎮 PS4 & Switch ### 👋 About me - Developer Advocate at [MuleSoft](https://www.mulesoft.com/) - Founder & Content Creator at [ProstDev](https://www.prostdev.com/) - Lead at [Women Who Code Monterrey](https://www.womenwhocode.com/monterrey) ### Latest Content ### GitHub Stats ### Connect ### MuleSoft Repos ``` ![Rendered README preview with the intro, emoji bullets and About me links plus section headers](../../assets/blog/how-to-create-a-readme-file-for-your-github-profile-part-2-markdown-basics-6.png) ### Commit your changes Once you feel happy with your changes, scroll-down until you see the “Commit changes” form and click the green button. ![GitHub Commit changes form with the default Update README.md message and green Commit button](../../assets/blog/how-to-create-a-readme-file-for-your-github-profile-part-2-markdown-basics-7.png) You can add a title to the commit, so you remember exactly which changes you did here, such as “Added intro and about me to `README.md`”. Note that if you don’t write anything, GitHub will automatically add the title “Update `README.md`,” so you don’t *have to* add a title/description to the commit, but it’s a best practice to keep in mind for your future projects. ![Commit changes form with a custom message Added intro and about me to README.md](../../assets/blog/how-to-create-a-readme-file-for-your-github-profile-part-2-markdown-basics-8.png) You’ll notice that there’s a green square on the right side of your repo that says this is a special repository. ![GitHub special-repository banner noting the README will appear on your public profile](../../assets/blog/how-to-create-a-readme-file-for-your-github-profile-part-2-markdown-basics-9.png) You should now be able to see this README file in your GitHub profile! If you have any issues, you can refer to the [official documentation here](https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-github-profile/customizing-your-profile/managing-your-profile-readme) or leave me a comment to troubleshoot! ### Recap - You need to create a repository with the same name as your GitHub username. - The hashtags in Markdown are used to specify a header or a title. “#” is the main title, and the following ones can be used as header 2 (##), header 3 (###), and so on. - Bulleted lists can be added in Markdown using the dash (-) character and leaving a space between the dash and the next character/word. - Hyperlinks can be added in Markdown using this syntax: [Text to display](link). For example, [ProstDev](https://www.prostdev.com/). - Remember to commit any changes done to the `README.md` file to your repository! Don’t miss any articles! Remember to subscribe at the bottom of the page to receive email notifications as soon as new content is published. ✨ -Alex ### GitHub repository [ProstDev - alexandramartinez](https://github.com/ProstDev/alexandramartinez) --- ## MuleSoft Runtime Fabric Deployed on Oracle Cloud Infrastructure (OCI) - Part 2: Mgmt & Operations Source: https://prostdev.com/post/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations | Published: Sep 21, 2021 | Category: Tutorials This is the second part of a series of articles focused on MuleSoft Runtime Fabric. In our [previous article](https://www.prostdev.com/post/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1), we’ve described how to deploy a full MuleSoft Runtime Fabric on top of Oracle Cloud Infrastructure. Now is the time to learn some mechanisms to manage that cluster. We are going to describe some tips about how to: - The process of deploying an application into the Runtime Fabric - The process to review the logs produced by the deployment process - Basic but useful commands to understand what is happening within the Runtime Fabric (namespaces, pods, secrets, etc) - Consolidate the log files into an external system (we are going to use Papertrail for our article) - Destroy pods - Test our applications - The usage of the rtfctl CLI Plus, a set of tips that are going to be useful for you while managing and maintaining your Runtime Fabric. Let’s start from the beginning. ## How to know if my cluster is up and running There are different ways to validate the health of your runtime fabric. Let’s elaborate on a couple of them: - Via Anypoint Platform console ([https://anypoint.mulesoft.com](https://anypoint.mulesoft.com/)) - Via the rtfctl CLI. We will get into some details with this one For the first option, you can get a very quick view by going into Runtime Manager -> Runtime Fabrics: ![Runtime Manager Runtime Fabrics view with summary and health detail blocks numbered 1, 2, 3](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-1.png) In the first block (circle number 1) you will have the option/link for Runtime Fabrics. In the second block (the purple one or number two in the image), you will get a summary of your Runtime Fabric, and finally, in the third block (the brown one or number 3) you will get a little bit of information like Application Status, Deployments, and Nodes. Let’s get a deeper look into block [#2](https://www.prostdev.com/blog/hashtags/2) (the purple one in the image): ![Runtime Fabric summary panel showing Active status, version 1.8.23, vCPU and memory](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-2.png) With this piece of information, you can validate: - That your cluster is Active. The other option is that it can be Degraded (we will elaborate more on that in the next paragraphs) or disconnected. The desired status is Active. - Then you can see the version of your Runtime Fabric, and it suggests that you upgrade to the latest one. In my case, I could upgrade to 1.10.0. But that will be part of another article in the upcoming future. - The number of vCPUs that you have for your Runtime Fabric and the Memory. - And a hyperlink to download the rtfctl CLI. If you are an operator and want to see how healthy your Runtime Fabric is, with this small section you can get it. Direct and simple. Now, sometimes things go in directions that we do not control that can make our cluster change its status. In that case in section [#3](https://www.prostdev.com/blog/hashtags/3) (the brown in the previous images) you can get some useful detail: ![Health Details panel: all systems operational with Appliance Status, Deployments and Nodes healthy](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-3.png) If something is wrong with the Application Status, your Deployments, or an event with your Nodes, you can get that detail from here. For example, the following image is from a Degraded cluster: ![Degraded cluster health detail warning that system components are degraded with failing nodes](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-4.png) As you can see, some of the errors that are causing the cluster to be in Degraded status are described in the different sections, in this case: in the Nodes section. For this error, something is wrong with our nodes, to the point that there is no status from them; they are probably down. Now let’s do something similar with RTFCTL CLI. ## RTFCTL CLI Since the very beginning of the computing era, the command-line interface (CLI) has been the common place to manage systems. But in this modern era, CLIs have become even more popular; and it is true to say that sometimes it is much more natural to use those CLIs rather than expecting a web-based console. For MuleSoft Runtime Fabric, there is an alternative for that, and that is called rtfctl. The official documentation is here: [https://docs.mulesoft.com/runtime-fabric/latest/install-rtfctl](https://docs.mulesoft.com/runtime-fabric/latest/install-rtfctl) If you want to install it, you have the following options: Windows: ```bash curl -L https://anypoint.mulesoft.com/runtimefabric/api/download/rtfctl-windows/latest -o rtfctl.exe ``` MacOS (Darwin): ```bash curl -L https://anypoint.mulesoft.com/runtimefabric/api/download/rtfctl-darwin/latest -o rtfctl ``` Linux: ```bash curl -L https://anypoint.mulesoft.com/runtimefabric/api/download/rtfctl/latest -o rtfctl ``` (Quick note: I normally install the rtfctl within the controller nodes of my Runtime Fabrics cluster.) Once you have it, make it available in your PATH (either Windows or Linux), and you can start using it. The available commands are: ![Table of rtfctl commands with descriptions, from appliance and apply to validate and version](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-5.png) (Taken from the official documentation [https://docs.mulesoft.com/runtime-fabric/latest/install-rtfctl](https://docs.mulesoft.com/runtime-fabric/latest/install-rtfctl)). As you can see, there are a lot of options, from getting to know the status of your cluster, to upgrading it. Let’s start from the most basic one: ```bash sudo rtfctl status ``` That command will return the following: ``` Using proxy configuration - (proxy "", no proxy "0.0.0.0/0,.local,0.0.0.0/0,.local,,0.0.0.0/0,.local,0.0.0.0/0,.local")" Using 'US' region transport-layer.prod.cloudhub.io:443 ✔ https://anypoint.mulesoft.com ✔ https://worker-cloud-helm-prod.s3.amazonaws.com ✔ https://exchange2-asset-manager-kprod.s3.amazonaws.com ✔ https://ecr.us-east-1.amazonaws.com ✔ https://494141260463.dkr.ecr.us-east-1.amazonaws.com ✔ https://prod-us-east-1-starport-layer-bucket.s3.amazonaws.com ✔ https://prod-us-east-1-starport-layer-bucket.s3.us-east-1.amazonaws.com ✔ https://runtime-fabric.s3.amazonaws.com ✔ configuration-resolver.prod.cloudhub.io:443 ✔ tcp://dias-ingestor-nginx.prod.cloudhub.io:5044 ✔ State: active Version: 1.1.1606942735-3f99c37 Nodes: controlrtf (11.0.0.5) : Status: healthy rtfworker1 (11.0.0.3) : Status: healthy rtfworker2 (11.0.0.4) : Status: healthy SERVICE HEALTHY agent true deployer true mule-clusterip-service true resource-cache true registry-creds true certificate-renewal true CERTIFICATE EXPIRES AT client 2022-03-12 02:08:43 +0000 UTC (199 days) rtfctl newer version 0.3.135 available; current version 0.3.102. Please use `rtfctl update` to update. ``` It is quite similar to what you get from the Anypoint Platform Web UI. Basically, it is telling us the status/state of our cluster, in this case: **Active**. And it also tells us the health of the controller and the workers, here: ``` Nodes: controlrtf (11.0.0.5) : Status: healthy rtfworker1 (11.0.0.3) : Status: healthy rtfworker2 (11.0.0.4) : Status: healthy ``` One crucial thing for this type of deployment (as described in our [previous](https://www.prostdev.com/post/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1) article) is the connectivity to the Control Plane. In a controlled environment where you have no limitations to connect to the internet, there is no problem to reach the control plane, but for enterprise deployments where you are behind a corporate firewall, you need to be sure that you can connect to all the different endpoints that the Runtime Fabric needs. Those endpoints are very well documented in the official documentation: [https://docs.mulesoft.com/runtime-fabric/1.3/install-port-reqs](https://docs.mulesoft.com/runtime-fabric/1.3/install-port-reqs) Specifically the section: Port IPs and Hostnames to Add to the Allowlist. As explained in our previous article, you must have connectivity to all those endpoints (depending on your region). But, sometimes, your cluster is not able to reach those endpoints, and one way to validate that it is running is the following command: ```bash sudo ./rtfctl test outbound-network ``` And the output is this: ``` Using proxy configuration - (proxy "", no proxy "0.0.0.0/0,.local,0.0.0.0/0,.local,,0.0.0.0/0,.local,0.0.0.0/0,.local")" Using 'US' region transport-layer.prod.cloudhub.io:443 ✔ https://anypoint.mulesoft.com ✔ https://worker-cloud-helm-prod.s3.amazonaws.com ✔ https://exchange2-asset-manager-kprod.s3.amazonaws.com ✔ https://ecr.us-east-1.amazonaws.com ✔ https://494141260463.dkr.ecr.us-east-1.amazonaws.com ✔ https://prod-us-east-1-starport-layer-bucket.s3.amazonaws.com ✔ https://prod-us-east-1-starport-layer-bucket.s3.us-east-1.amazonaws.com ✔ https://runtime-fabric.s3.amazonaws.com ✔ configuration-resolver.prod.cloudhub.io:443 ✔ tcp://dias-ingestor-nginx.prod.cloudhub.io:5044 ✔ ``` With that, you can validate if your Runtime Fabric is fully connected to the control plane. If any of those are not responding, you will notice that your cluster will change into a **Degraded** state. Some of those endpoints can be obvious, for example, [https://anypoint.mulesoft.com](https://anypoint.mulesoft.com/) is the one serving most of the calls (API Manager and Exchange to give you an example). But some others like the amazonaws.com endpoints are used as the docker image repository. What we are trying to explain is that all of them have a specific purpose and will lead the Runtime Fabric to different behaviors. Another useful command is the one used to update your MuleSoft license.lic file. Remember that MuleSoft is a subscription-based platform and every year a new license.lic file is generated. That means that every year you need to update it in your Runtime Fabric. In order to do that, you can execute these two commands: 1. Transform the content of the license.lic file into base64 format with this command: ```bash base64 -w0 license.lic ``` 2. With the base64 string execute this: ```bash sudo ./rtfctl apply mule-license BASE64_ENCODED_LICENSE ``` 3. And if you want to check/validate the license: ```bash sudo ./rtfctl get mule-license ``` ## Deploy applications to the Runtime Fabric The deployment process is no different from what you have for CloudHub or even for a Standalone, from the Runtime Manager perspective. There are key differences to understand, not only for the deployment configuration but the dynamics that happen during and after the deployment. In this section, we will go into those differences. Let’s keep it simple, and let’s deploy an application using Runtime Manager from [https://anypoint.mulesoft.com](https://anypoint.mulesoft.com/). The name of the application is myRTFapp. And it contains a very simple structure: ![Anypoint Studio myrtfapp flow with Listener and Logger, Logger message set to payload](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-6.png) Our listener is configured to serve on port 8081 (HTTP). Since we are on Runtime Fabric, your listener must be listening on 8081 or 8082. The logger is going to be valuable for what we are going to explain in the next paragraphs. But first, let’s go through the deployment process. From the Runtime Manager perspective, the process itself is similar to CloudHub or Standalone Hybrid Model. You have to go into Runtime Manager and deploy the application, selecting your Runtime Fabric as your target: ![Runtime Manager Applications view highlighting the Deploy application button](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-7.png) And then: ![Deploy Application form selecting the ocigen2rtf Runtime Fabric as deployment target](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-8.png) Once you select an RTF-based deployment target, you will get the following different options that you can decide to configure/use while making the deployment: ![RTF deployment options: runtime version, replicas, cluster mode, rolling update and resource allocation](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-9.png) - You can decide the runtime version for your applications. That means that on top of Runtime Fabric you can have different applications using different MuleSoft Runtime versions. - Replicas: This is the desired number of replicas for your application. That means that your application is replicated and therefore can distribute its load along with those replicas. But also, that means that the Runtime Fabric will have to ensure to have those replicas up and running since that is the desired state for your application. That is one of the benefits of this deployment model. - You can run your application in cluster mode. Clustering is one of the options for MuleSoft Runtime, and that is possible to have in Runtime Fabric. If you check that box, your application will run in cluster mode. The checkbox will be enabled if you choose to have more than 1 replica for your application. That makes sense since in order to have a cluster you need at least two nodes. - Deployment Model: This is very useful when you are introducing a change to your application. The rollout will be incremental, updating replica by replica. This will translate into zero downtime for your application. - Reserved CPU, CPU Limit, and Memory. You can decide the amount of computing power your application needs. You can also configure specific properties for your application, just as you have it for CloudHub. And you can also configure the JVM properties for the Runtime. After you click on the **deployment** button several things happen behind the scenes: - The application/artifact is registered on Exchange. - A Docker-based image is uploaded to the Image Repository (AWS). - The Runtime Fabric creates the deployment using the image created in step [#2](https://www.prostdev.com/blog/hashtags/2). - The application gets deployed on top of the Runtime Fabric and a Pod with two containers is created: - One container holds the MuleSoft Runtime with your application. - The second container is a sidecar for the application, for monitoring purposes. The result at the Runtime Fabric level is the following: ![kubectl pod list showing myrtfapp pod in Init:0/1 then PodInitializing status](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-10.png) It starts initializing your pod, in this case: myrtpapp-79b9b4c488. Then it will transition into: ![kubectl pod list showing myrtfapp containers reaching 1/2 then 2/2 Running over 32 seconds](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-11.png) As you can see it started one of the containers (after 8 seconds) and then the second container (after 32 seconds). Those are the two containers that we explained in the previous paragraph. Once you see that both containers are READY and the STATUS is Running, your application is ready to be used. And from the Anypoint Web Console, you have: ![Anypoint console myrtfapp detail showing Running status, 1/1 replicas started on ocigen2rtf target](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-12.png) Up to this point, your deployment process is done. In the next section, we will learn how to get the logs from a particular application. ## Get the log files from my applications and the RTF platform The application is now running in your RTF. It is deployed in a pod and is serving requests. But before getting into the log files, there is something relevant to notice and is related to the RTF underlying infrastructure: Kubernetes. If you are familiar with Kubernetes, one of the main concepts and elements is the namespaces, which are a logical way to organize your applications. For our RTF-based deployment model, that is important, since a namespace will represent an Anypoint environment (for example Sandbox, Production). Therefore, when you deploy an application since you decide in which environment to deploy it, you need to know the ID for that environment in order to perform some actions at the RTF level. > [!NOTE] > the following commands are executed from the controller node. Let’s explain it with some commands: ```bash kubectl get namespaces ``` This will return the list of namespaces for your RTF: ![kubectl get namespaces output listing the Sandbox environment UUID plus rtf and kube namespaces](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-13.png) The first namespace that was returned is: 823d7ade-927a-4b2a-9f1e-2b56b06a6ce0. Which is our SANDBOX environment at Anypoint. How do I know that? Let’s get it from the Web Console. From the API Manager menu: ![API Manager Sandbox administration with the environment-info button highlighted](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-14.png) Click on the desired environment, and then click the button with the exclamation mark. It will return the environment information: ![Environment Information panel showing Sandbox name and its environment ID matching the namespace](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-15.png) Among other things, the environment ID, which matches with the namespace that we’ve explained. An RTF can be associated with one or more environments, it is up to you how you want to organize your applications and environments. That option is available from the RTF configuration page: ![RTF Associated Environments tab with Sandbox checked to associate it with the Runtime Fabric](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-16.png) In my case, Sandbox is the only environment associated with my RTF, and that is why we just see that namespace. With this explanation in mind, we will learn how to get the logs from our applications. 1. The first step is to get the namespace of the environment where you deployed your application. 2. With the namespace ID, we will execute: ```bash kubectl get pods -n ``` In our case: ```bash kubectl get pods -n 823d7ade-927a-4b2a-9f1e-2b56b06a6ce0 ``` This will return the list of created pods in that namespace. Since every pod is a MuleSoft Application, it will return the list of applications: ![kubectl get pods for the Sandbox namespace listing four running RTF application pods](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-17.png) In this case, we have 4 applications running on top of our RTF. You will notice that it is the very same list and status that you get from the web console: ![Anypoint console application list showing the same four apps Running on the ocigen2rtf server](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-18.png) 3. Now that you have the list of applications, you can get the log files from a specific application using the next command: ```bash kubectl logs deploy/ -n -c app ``` In our case: ```bash kubectl logs deploy/myrtfapp -n 823d7ade-927a-4b2a-9f1e-2b56b06a6ce0 -c app ``` Please notice that the app-name is the name that you decided to use when you deployed the application. Which happens to be the name of the POD, but without the random ID created by Kubernetes. 4. After executing the command, you will get the runtime log, which is the normal log file from your MuleSoft runtime: ![Terminal showing the Mule runtime application log retrieved via kubectl logs](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-19.png) 5. But if you want to have the log file open while you make a request to your application in order to see what it’s writing, you can use the following command (which is like a tail -f): ```bash kubectl logs deploy/ -n -c app --follow ``` In our case: ```bash kubectl logs deploy/myrtfapp -n 823d7ade-927a-4b2a-9f1e-2b56b06a6ce0 -c app --follow ``` With that command, the log file will be in your terminal. If you make a request you will see what the log reports. 6. For example, remember that we mentioned that our sample app had a logger processor that will print the whole payload? Let’s make a call to it and see the result in our log file. Let’s first get the IP of the Pod (in this case I will make a call directly to the pod, just for demo purposes, but in real life, you will have to do it using your ingress controller). To get the IP of the pod, execute this: ```bash kubectl get pods -n -o wide ``` In our case: ```bash kubectl get pods -n823d7ade-927a-4b2a-9f1e-2b56b06a6ce0 -o wide ``` And the result is: ![kubectl get pods -o wide output revealing each pod's IP address and node](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-20.png) The IP is the range of the CIDR block or the pods' network that we explained in our previous [article](https://www.prostdev.com/post/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1). Our application is being served at IP: 10.244.15.66 and port 8081. And the resource path: /myrtf Let's send a POST with some payload to it: ```bash curl -X POST http://10.244.15.66:8081/myrtf -d '{"message":"Hello RTF"}' ``` In our log file: ![Followed log showing the Logger printing the Hello RTF payload after the curl POST request](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-21.png) That is from the application perspective, but what about the logs of the agent that performs the deployments into RTF? Let's imagine that something went wrong during the deployment, and you want to learn what happened. You can use this command to get the logs of that agent: ```bash kubectl -n rtf logs -f deployment/agent ``` You will notice that when you execute that command, a lot of log entries will be retrieved. You can either save them to a file and then do some research on it, or consolidate all your RTF log files (both the platform and applications) into Splunk, Elasticsearch, or a Syslog platform. We will elaborate a little bit more in the next section. Other useful commands, but now from the rtfctl perspective, are the ones that are related to how to retrieve: - HEAP dumps from your application. - THREAD dumps from your application. Sometimes it is useful to get into that level, and you can achieve it using the following commands: ```bash sudo ./rtfctl heapdump sudo ./rtfctl threaddump ``` For example: ```bash sudo ./rtfctl threaddump myrtfapp ``` The output is: ![Terminal output of rtfctl threaddump showing JVM thread stack traces for myrtfapp](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-22.png) (It is much larger, but we just want you to see that it is a normal thread dump from your app) ```bash sudo ./rtfctl heapdump myrtfapp dump.dmp ``` ![Terminal showing rtfctl heapdump writing the myrtfapp heap to dump.dmp](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-23.png) At this point, you may be wondering if it would be easier to consolidate most of the log files of our platform into a single place. That is completely possible with RTF and is very straightforward to configure. Just go to your RTF at the Anypoint Platform Web Console: ![ocigen2rtf Runtime Fabric page with the Log Forwarding tab highlighted](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-24.png) In that tab you will see this: ![Log Forwarding service dropdown listing Azure, Elasticsearch, Graylog, Splunk, CloudWatch and Syslog](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-25.png) As you can see, there are several options for this. In our case we will use Syslog: ![Syslog external configuration form (host, port, udp mode, rfc5424) and which logs to forward](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-26.png) The green square is for the connectivity information you need to input and is related to your SYSLOG server. And also for the format of the syslog, in our case, we will use the format rfc5424. The orange square is for you to decide which logs you want to send to the syslog server. You will notice that there are several options. Decide on what fits better for you. In our example, we will send everything to the PaperTrail syslog UDP server. Once you’ve configured it you can send a text message: ![Syslog forwarding enabled, configuration applied, with a Send a test message link](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-27.png) Once you click on it at the top right corner you will get: ![Green toast notification reading Message sent, visit Syslog to verify the connection](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-28.png) And after that, you can see the events being sent. Most of these events are from the platform itself: ![Papertrail Events view streaming incoming RTF platform log entries](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-29.png) But we can filter it with the name of our application; myrtfapp: ![Papertrail search bar filtering all systems by the term myrtfapp](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-30.png) Let’s send a request to our app and we will see the log file in Papertrail: ```bash curl -X POST http://10.244.15.66:8081/myrtf -d '{ "array": [ 1, 2, 3 ], "boolean": true, "color": "gold", "null": null, "number": 123, "object": { "a": "b", "c": "d" }, "string": "Hello World" }' ``` And at PaperTrail: ![Papertrail showing myrtfapp Logger output with the JSON payload forwarded via Syslog](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-31.png) Now let’s see how to get inside a pod. ## How to get inside a pod Sometimes you will have to get within a pod in order to do a particular activity. For example: to test connectivity to an external resource. Or just because you are curious, and you want to see what is inside of it. In order to get into the pod, execute this command: ```bash kubectl exec --stdin --tty -n -- /bin/bash ``` In this case, we are using the entire pod name. For example: ```bash kubectl exec --stdin --tty myrtfapp-79b9b4c488-4dm7v -n 823d7ade-927a-4b2a-9f1e-2b56b06a6ce0 -- /bin/bash ``` Once you execute this, the command prompt will be returned and you will be inside the pod: ![kubectl exec opening a bash shell inside the myrtfapp pod at /opt/mule](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-32.png) As you can see in the image, I am inside the pod. If I execute: ```bash ps -fea ``` I will see the java process executing my MuleSoft Runtime: ![ps -fea run inside the pod showing the Java process running the Mule runtime](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-33.png) You can move to the apps folder and see your application: ![ls of /opt/mule/apps inside the pod listing the myrtfapp deployment and its anchor file](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-2-mgmt-operations-34.png) You cannot write or modify anything. You do not need it, that is the beauty of this. If you need to change something, for example, JVM properties. You will do it from the console, and a new POD will be created with such a configuration. ## Conclusion and next steps With this article, we finalized this small series of two articles about RTF on top of Oracle Cloud Infrastructure. You’ve learned some very basic but useful commands. You now have a glimpse of how the underlying RTF infrastructure works, and how it is related to Kubernetes. We will continue writing about RTF. Our next article will be targeted at the Operation Center web console, which is also very useful for management purposes. We will also explain the upgrade process of an RTF. Stay tuned! --- ## How to create a README file for your GitHub profile, part 1: Intro to Git, GitHub, and README files Source: https://prostdev.com/post/how-to-create-a-readme-file-for-your-github-profile-part-1-intro-to-git-github-and-readme-files | Published: Sep 14, 2021 | Category: Guides *GitHub repository with the final `README.md` file can be found at the end of the post.* Having a [GitHub](https://github.com/) account in today’s world is a **must-have** for any software developer. In this series of posts, I’ll show you how to create a striking `README.md` file to attach to your GitHub profile. If you want to see an example of the final result, take a look at [my GitHub profile](https://github.com/alexandramartinez)! *Note: my GitHub profile's README file may have changed over time. To see the exact same code demonstrated throughout this series, check out* [this repository](https://github.com/ProstDev/alexandramartinez)*.* ## Git vs. GitHub First of all, if you’re not familiar with [Git](https://en.wikipedia.org/wiki/Git), it’s a technology that is widely used by almost every software project nowadays. The main feature of Git is the ability to keep your code in “repositories” and improve the collaboration between several developers who may be working on the same code at the same time. Think of it as a [Google Doc](https://workspace.google.com/products/docs/) for programming. Here are some sites to learn about Git if you’ve never used it before: [Git Documentation](https://git-scm.com/doc), [Learn Git Branching](https://learngitbranching.js.org/). Now, a common misconception is thinking that **GitHub** is the same as **Git**. While GitHub is based on Git (for the majority), they’re two different things. GitHub is what you see (front-end): the user interface, the buttons, the menus, the images, etc. While Git is the technology used underneath (back-end): the ability to create and manage repositories, commit changes, share your code, collaborate with others, etc. Other sites/platforms are also based on Git, for example, [BitBucket](https://bitbucket.org/product) or [GitLab](https://about.gitlab.com/). So why GitHub? In my experience, almost all developers I’ve met keep their repositories on this platform. A lot of social networking sites have integrations to add your GitHub profile to your list of links or have the ability to embed your repositories in a post (for example, [Polywork](https://www.polywork.com/devalexmartinez/highlights/897f5203-0527-4334-9f17-ce1591129cc4)). And in my personal opinion, I think it’s easier to use. The UI/UX is beautiful, it has a ton of features that they keep improving regularly, and it’s 100% free! ## About README.md files Whenever you create a new repository, it is a best practice to create a `README.md` file at the root of your project. This file needs to explain the basics of your project: what it is, what it does, and how to install/use it. It’s basically a “welcome” page for new developers that want to use your code. Here’s a very nice example of a `README.md` file for the [Official Jenkins Docker image repository](https://github.com/jenkinsci/docker#readme). GitHub has the feature of attaching a README file to your profile to introduce yourself and/or give a summary of your GitHub page for new developers visiting your profile. You are free to personalize this file with whatever information you want to show to the public. Some people use images or GIFs, embed videos, add links, or give [CTAs](https://en.wikipedia.org/wiki/Call_to_action_(marketing)) here. Think of it as a tiny website where you get to show off your work, your hobbies and let people know who you are and what you do. ## Inspiration When I started creating my README, I was inspired by [this GitHub profile](https://github.com/victoria-lo). Then I noticed she has a repository with a compilation of some other [awesome GitHub profiles](https://github.com/victoria-lo/awesome-github-profiles), which inspired me even more! I didn’t know about a lot of the features that could be added to README files; I discovered them after I saw all these great profiles. ## Before reading the rest of the series If you’re new to Markdown, they may seem a bit complicated or overwhelming. But not to worry! In this series of posts, I’ll guide you through all the steps I did to get my README file up and running, even if you’re new to all this. If you’re new to Git, I do recommend you first get a sense of the basic terminology such as repository, blob, commit, local/remote, branch, etc. I found [this article](https://www.javatpoint.com/git-terminology) on Google. It may help you to have a rough idea of the theory. You don’t really need to know Git in order to follow the series. You can just follow the steps and still get the final outcome. However, it may seem a bit confusing if you don’t understand the concepts behind it. ## Recap - GitHub is based on Git, but they’re not the same thing. Git is the back-end part, and GitHub is the front-end part. - `README.md` files are used to explain the basics of a repository: what it does, how to install, and how to use it. - In GitHub, you can attach a README to your profile to welcome your readers. Don’t miss any articles! Remember to subscribe at the bottom of the page to receive email notifications as soon as new content is published. ✨ -Alex ## GitHub repository [ProstDev - alexandramartinez](https://github.com/ProstDev/alexandramartinez) --- ## Notification center for new versions of connectors/modules in Anypoint Studio 7.10.0 Source: https://prostdev.com/post/notification-center-for-new-versions-of-connectors-modules | Published: Aug 31, 2021 | Category: News Let’s do a quick and short blog post! This post is about the new Anypoint Studio 7.10.0 version which has been released this past July 29th, 2021. In this link you will find all the details of this new release: [https://docs.mulesoft.com/release-notes/studio/anypoint-studio-7.10-with-4.3-runtime-release-notes](https://docs.mulesoft.com/release-notes/studio/anypoint-studio-7.10-with-4.3-runtime-release-notes) ## New features summary Here is a summary of the new changes introduced in this new version, you will find the same summary within your Anypoint Studio installation at *Help > What’s new in Anypoint Studio?* **Introducing Metadata Assistant - New** MuleSoft is making it easier to work with metadata across flows and sub-flows. You can now extract and propagate metadata across sub-flows with a few clicks through a new wizard, where previously this was a laborious manual process. **Notification center for new versions of connectors - New** Get recommendations when new connector versions are available so you are on the latest innovation. **Refactor rename for flows - New** Now it will be easier to rename flow references. Changes on a flow name will be propagated automatically across flow references. **Themes - Improvement** Theme selection got smarter than ever. For new workspaces, it will automatically match your Operating System appearance preference selecting the Light or Dark theme accordingly. ## Notification Center for new versions of connectors Let’s talk about the new Notification Center for new versions of connectors which is a very cool feature. Have you ever been asked to update the dependencies of your MuleSoft project? Have you ever noticed that the modules or connectors of your projects are older versions? Have you had problems with not updating your code to the latest versions? The new notification center solves most of these problems. To use it you simply have to do the following. Let’s say you are developing a MuleSoft application where you need to use the database and file connectors. You will see in the `pom.xml` file those two dependencies. Similar to this: ![pom.xml dependencies showing the Database connector 1.10.0 and File connector 1.3.0](../../assets/blog/notification-center-for-new-versions-of-connectors-modules-1.png) As you can see, your project is using version 1.10.0 for the database connector and 1.3.0 for the file connector. Are those the latest version of the connectors? Could this cause an issue in your application? Do you want to verify and update them to the newest versions? Let’s check it out! Right-click on your project > Manage Dependencies > Manage Modules: ![Studio right-click project menu with Manage Dependencies then Manage Modules highlighted](../../assets/blog/notification-center-for-new-versions-of-connectors-modules-2.png) Automatically, Anypoint Studio will connect with Anypoint Exchange and will fetch all the alternative versions for your connectors: ![Modules properties listing connectors with blue update dots while fetching alternative versions](../../assets/blog/notification-center-for-new-versions-of-connectors-modules-3.png) The blue dots indicate that there are available new versions for those connectors. Then, you need to click on each connector, click on the Update Version button, and finally on Apply and Close button. ![Modules panel with "New version available" for Database and the Update version and Apply buttons](../../assets/blog/notification-center-for-new-versions-of-connectors-modules-4.png) You can confirm that the `pom.xml` file is automatically updated with the new dependencies versions. ![pom.xml reopened in Studio showing the connector dependencies updated to their new versions](../../assets/blog/notification-center-for-new-versions-of-connectors-modules-5.png) Finally, you can confirm that the versions of those dependencies are the latest for each connector at the Anypoint Connector Release Notes page: [https://docs.mulesoft.com/release-notes/connector/anypoint-connector-release-notes](https://docs.mulesoft.com/release-notes/connector/anypoint-connector-release-notes) ![Latest version of the Database Connector](../../assets/blog/notification-center-for-new-versions-of-connectors-modules-6.png) ![Latest version of the File Connector](../../assets/blog/notification-center-for-new-versions-of-connectors-modules-7.png) Easy, right? Enjoy and happy coding! --- ## Recommendation: Logitech MX Ergo Advanced (Plus) Wireless Trackball Mouse Source: https://prostdev.com/post/recommendation-logitech-mx-ergo-advanced-plus-wireless-trackball-mouse | Published: Aug 10, 2021 | Category: Opinion *Spanish version / versión en Español:* [Recomendación: Logitech MX Ergo Advanced (Plus) Wireless Trackball Mouse](https://www.alexandramartinez.world/post/recomendacion-logitech-mx-ergo-advanced-plus-wireless-trackball-mouse). I bought this mouse less than 24 hours ago, and I’m already completely in love with it. I’m honestly even a little mad at Logitech for not sending me an email with the marketing for this product as soon as it came out, *LOL*! ## My previous mouse Before starting with why I love my new purchase, let me give you a bit of background on my previous mouse. It was a [Logitech M570](https://amzn.to/3sTtftz) (also a wireless trackball mouse). I was pretty comfortable with it, but I always thought it needed more buttons. However, the newer versions still had pretty much the same number of buttons, so I never thought of upgrading it...until now. If you don’t know what a trackball mouse looks like, here it is: ![Logitech M570 Wireless Trackball Mouse.](../../assets/blog/recommendation-logitech-mx-ergo-advanced-plus-wireless-trackball-mouse-1.png) Why do I love these? - I don’t need a mouse pad. - I don’t need to move my hand; I just move my thumb. - Less desktop space is needed. - It can be used on almost any surface. I say it needs more buttons because you basically have (1) the regular left-click, (2) right-click, and (3) scroll wheel button; *plus,* you can find the two buttons between the trackball and the left-click button. ![Logitech M570 Wireless Trackball Mouse’s 5 programmable buttons.](../../assets/blog/recommendation-logitech-mx-ergo-advanced-plus-wireless-trackball-mouse-2.png) I wanted to program several buttons to do some of my more common tasks - *I’ll mention them later in the post.* Unfortunately, I was only able to use buttons 3, 4, and 5 for my custom functions. There are a lot of mice with more than 5 programmable buttons, but I wanted it to be specifically a trackball mouse. This brings me to my new toy! ## Logitech MX Ergo Advanced (Plus) Wireless Trackball Mouse ![Logitech MX Ergo Advanced Wireless Trackball Mouse’s 6 programmable buttons.](../../assets/blog/recommendation-logitech-mx-ergo-advanced-plus-wireless-trackball-mouse-3.png) [This mouse](https://amzn.to/3eVb5z6) has **9** buttons in total! The picture may be misleading because it’s only pointing to the *6 programmable* buttons, but it’s not counting the left-click, right-click, and the small button in the middle that says 1 and 2. These 3 buttons will always maintain their specific functions and cannot be changed. I’m sure you’re curious as to what I have programmed these 6 buttons to do. Here it goes: - Left-top button: Delete. - Left-bottom button: Enter/Return. - Scroll wheel left button: Move to the desktop to the left (for Mac computers). - Scroll wheel middle button: Screen capture. - Scroll wheel right button: Move to the desktop to the right (for Mac computers). - Button over the trackball: Maximize/minimize window. Before I got an additional keyboard, I didn’t have to use numbers 3 and 5 that often. I could just use my Macbook’s trackpad to swipe left/right and change my desktop. However, after acquiring the keyboard, this became a bit more work. I had to either reach out to the laptop’s trackpad with my left hand or use the keyboard shortcut with my right hand (which I had to switch between the keyboard and the mouse). Neither option was good enough for me. ![Reference picture of my current desktop setting.](../../assets/blog/recommendation-logitech-mx-ergo-advanced-plus-wireless-trackball-mouse-4.jpg) With my previous mouse, I had to start using the two additional buttons (4 and 5) to move to the left/right desktop. So, my previously programmed keystrokes (Delete and Enter/Return) were no longer in the picture. This, again, was not good enough for me. Now I get to have all my 4 favorite tasks PLUS 2 more buttons that make my life easier! ## Is that all? Of course not! This mouse is a MASTERPIECE! But the buttons were my biggest need. Also, the M570 requires one AA battery, but the MX Ergo can be recharged with a USB. By the way, remember the middle button with the 1 and 2 numbers underneath it that you could not program? ![Close-up of the mouse's 1 and 2 button for switching between two paired computers](../../assets/blog/recommendation-logitech-mx-ergo-advanced-plus-wireless-trackball-mouse-5.png) Well, that button is for alternating between 2 different computers with the same mouse! This is incredibly useful if you’re using several systems at the same time. It can be annoying to have to use different hardware for the same purpose. I tested this functionality, and it doesn’t even have a lag to it. Once you push the button, the other computer can be used immediately — no need to wait for even 1 second. It still has more great specs, but they weren’t *that* important for my day-to-day. You can read more about them [here](https://www.logitech.com/en-ca/products/mice/mx-ergo-wireless-trackball-mouse.html). Finally, I leave you this quick 30-second video for a preview! --- ## MuleSoft Runtime Fabric Deployed on Oracle Cloud Infrastructure (OCI) - Part 1 Source: https://prostdev.com/post/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1 | Published: Aug 3, 2021 | Category: Tutorials MuleSoft is so flexible and modern in such a way, that for deployment models where the customer needs to have control over the infrastructure, and CloudHub is not an option (maybe regulations concerns) it offers on premise deployments such as: - MuleSoft Standalone - MuleSoft Runtime Fabric - MuleSoft Private Cloud Edition The first two are a hybrid model, where the control plane remains in the cloud (MuleSoft) and the runtime plane is deployed on premise or in a computer on top of a cloud provider (Google, AWS, Oracle, Microsoft, Digital Ocean, etc). For Standalone, it’s the common MuleSoft runtime running directly at the OS level, which supports clustering, grouping of runtimes, etc. There is nothing extra on this type of deployment, it is basically to download the software, unzip it and run it. I am being practical right here, but in essence that is what you have to do in order to have it running on your own hardware. In this case the responsibility to maintain, patch, upgrade, operate the runtime is at the customer end; the control plane remains a MuleSoft responsibility, but ultimately where the workloads are running (runtime plane) is all at customer’s. Private Cloud Edition, is the option where you have both the Runtime Plane and the Control Plane at your end; you are in charge of maintaining everything. You install it, you patch it, you upgrade it, you monitor it, etc. But you have most of the capabilities at your disposal, similar to CloudHub; there are some differences, but in general it is very compatible with what you get in CloudHub. You can decide how much compute to assign to a specific MuleSoft application, or the amount of replicas, in a very similar CloudHub fashion. Then you have MuleSoft Runtime Fabric (RTF), which is the option we want to elaborate through this article. Runtime Fabric, as previously mentioned, is a hybrid deployment model. You still have the Control Plane in the cloud but the workloads (runtime plane) are running at the customer end. The main thing here with RTF is that the infrastructure that is behind the scenes running the fabric is a Kubernetes-based implementation (Kubernetes [https://kubernetes.io](https://kubernetes.io/)). But before thinking that this may be very complicated, continue reading this article where you will realize that some of the complexity that is coming to your mind right now, is going to get reduced. Let us start with the installation process, which is always time consuming because of the pre-requisites that we need to fulfill every time we install something on premise. With RTF is no exception, you need to have all the pre-requisites in place in order to start installing. But the good news is that it is very well documented. You will identify two ways of installing RTF: 1. **On top of an existent Kubernetes cluster that you already have**. For example, on top of AWS, GKE, AKE. This option is the best one if you want to leverage from what you already have in terms of Kubernetes. The installation documentation is here [https://docs.MuleSoft.com/runtime-fabric/latest/install-self-managed](https://docs.mulesoft.com/runtime-fabric/latest/install-self-managed). ![Runtime Fabric on Self-Managed Kubernetes.](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-1.png) 2. **Manual Installation**. This option is where you want to install RTF on top of your hardware, either if it is at your datacenter or on the compute at your cloud provider. This article is based on the manual installation and can be useful for you in case you do not have an existent Kubernetes cluster and want to test RTF. This manual installation is also useful to understand the dynamics behind RTF and is a very good option for customers who don't have a current Kubernetes cluster, but want to have a flexible/dynamic deployment model. The official documentation is this one: [https://docs.MuleSoft.com/runtime-fabric/1.9/install-manual](https://docs.mulesoft.com/runtime-fabric/1.9/install-manual). ## Infrastructure Summary I am going to use Oracle Cloud Infrastructure to explain the pre-requisites and part of the installation process. Why Oracle Cloud Infrastructure? I think it is a very good option for this demo and very simple to use. What I am using from Oracle Cloud Infrastructure (OCI) is the following: - Compute Instances - Storage - Load Balancers - Virtual Networks and Subnets - Security access rules (firewall) In this case I will use a three nodes RTF cluster, where: - One of the nodes is acting as the Controller - Two nodes are going to represent the Workers ## Compute & Storage ![OCI Compute Instances list: controlRTF, rtfworker2 and rtfworker1 all running VM.Standard.E2.2](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-2.png) Operative system can be any of the following: - Red Hat (RHEL) v7.4, v7.5, v7.6, v7.7, v7.8, v7.9, v8.0, v8.1, v8.2, v8.3 - CentOS v7.4, v7.5, v7.6, v7.7, v7.8, v7.9, v8.0, v8.1, v8.2, v8.3 - Ubuntu v18.04 (using Runtime Fabric Appliance version 1.1.1625094374-7058b20 or later) I am using CentOS. Within those instances I need to take care of this: - Disable swap memory. - Ensure that chrony (system clocks) status is in sync. I have provisioned storage to attach it to those compute instances: ![OCI Block Volumes list with worker, controller Docker and controller etcd volumes](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-3.png) The storage is distributed in the following way: ![Table mapping each compute node to its storage volume, purpose and size](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-4.png) The storage is just presented to the compute instance but is not formatted. The installation process will take it from there, format and attach it in the way RTF needs it. The storage for the controller, is for the following purpose: - A minimum of 60 GiB dedicated disk with at least 3000 provisioned IOPS to run the etcd ([https://etcd.io/](https://etcd.io/)) distributed database. This translates to a minimum of 12 Megabytes per second (MBps) disk read/write performance. - A minimum of 250 GiB dedicated disk with 1000 provisioned IOPS for Docker overlay and other internal services. This translates to a minimum of 4 MBps disk read/write performance. For the workers: - A minimum of 250 GiB dedicated disk with at least 1000 provisioned IOPS for Docker overlay and other internal services. This translates to a minimum of 4 MBps disk read/write performance. - Having 250 GiB ensures there is enough space to run applications, cache docker images, provide temporary storage for running applications, and provide log storage. In the local disk: - A minimum of 80 GiB dedicated disk for the operating system. - A minimum of 20 GiB for /tmp directory. - A minimum of 8 GiB for /opt/anypoint/runtimefabric directory. - A minimum of 1 GiB for /var/log/ directory. ## Networking Regarding networking, I have this VCN: ![OCI Virtual Cloud Networks list showing the rtfvcn network with CIDR block 11.0.0.0/16](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-5.png) I have a large CIDR Block 11.0.0.0/16 but you can use a smaller one. Just keep in mind the following: > The pod CIDR block must not overlap with IP addresses that pods or servers use to communicate. If services within the cluster or services that you installed on nodes need to communicate with an IP range that overlaps the pod or service CIDR block, a conflict can occur. If a CIDR block is in use, but pods and services do not use those IP addresses to communicate, there is no conflict. If you deploy more than one cluster, each cluster can reuse the same IP range, because those addresses exist within the cluster nodes, and cluster-to-cluster communications is relayed on the external interfaces. *Taken from the official documentation:* [https://docs.MuleSoft.com/runtime-fabric/1.9/install-prereqs](https://docs.MuleSoft.com/runtime-fabric/1.9/install-prereqs)*.* Also, within that VCN, I have a couple of subnets: ![OCI Subnets list with a private 11.0.1.0/24 subnet and a public 11.0.0.0/24 subnet](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-6.png) One is private and one public. For our scenario, in order to have access to the compute instances, all VMs are part of the Public subnet. Here is the summary: ![Table of the three nodes' primary VNIC details: private IPs, subnet and hostnames](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-7.png) My computer instances have the following characteristics: ![Compute instance specs: VM.Standard.E2.2, 2 GHz AMD EPYC 7551, 2 CPUs, 16 GB RAM](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-8.png) All compute instances are interconnected and need to be visible among each other though several ports and protocols. Those ports and protocols, in my case in Oracle Cloud Infrastructure, are reflected through a Security List. Which would be the equivalent to the configuration you have to put in place in your firewall. The summary is described in the following table: ![Security list table of required ingress rules: source, protocol, port range and description](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-9.png) As mentioned, that was all defined in my Default Security List: ![Default Security List for rtfvcn page showing 21 ingress rules](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-10.png) Another relevant thing is that both controllers and workers need to have access to the Control Plane, and therefore to the Internet to reach certain hostnames. Keep in mind that we are working with a hybrid model and the runtime plane needs access to the control plane for several purposes: - Monitoring - Access to the image repository - Access to Runtime Manager - Access to API Manager - Etc Within our infrastructure (OCI) we have an Internet Gateway that gives us access to the internet: ![rtfvcn details page with its Internet Gateway listed under the VCN](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-11.png) In my case with that simple Internet Gateway I get access to the internet from within my compute instances. In your case you may control this, and just enable access to the specific list of locations. That specific list is the following: ![Table of required outbound hostnames on port 443 for US and EU control planes](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-12.png) ![Continued hostnames table covering the deprecated port 5044 Lumberjack endpoints](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-13.png) ![Hostnames table for Anypoint asset, Kubernetes charts and Docker repositories on port 443](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-14.png) ![Hostnames table for Anypoint Exchange and Runtime Fabric Docker repository endpoints](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-15.png) ![Hostnames table for Runtime Fabric Docker image delivery and Configuration Resolver endpoints](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-16.png) Depending on if you are in the US or EU control plane, you will need to reach some of those locations. All this information is here: [https://docs.MuleSoft.com/runtime-fabric/1.9/install-prereqs](https://docs.mulesoft.com/runtime-fabric/1.9/install-prereqs) try to follow the official documentation to avoid any error. ## Installation Process Again, this article does not replace the official documentation. We are just trying to give you a glimpse of the whole process and share with you that the whole installation process is very accessible. And you can execute within hours, trust us. The first step is to declare in the Anypoint Platform web console that you want to create an RTF. That is a very simple step, just go here: ![Runtime Manager Runtime Fabrics page with the Create Runtime Fabric button highlighted](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-17.png) Select your environment, and then click on the Create Runtime Fabric button. A similar screen as the following will appear: ![Create Runtime Fabric dialog with install targets and the VMs or bare metal option highlighted](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-18.png) Give a name and select the VMs or bare metal radio button. Once you do that, this screen will appear: ![Runtime Fabric setup steps: install overview video, Download files button and activation data](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-19.png) Look at step 2 and step 3. Step 2 will download the required installation files, and Step 3 is your activation data for your installation. Please copy that and put it in a safe place. Once you download the file (a similar name to this rtf-install-scripts-20210709-b48dec6.zip), upload it to the three servers (controller, worker1, worker2). Now unzip it: ```bash mkdir -p ./rtf-install-scripts && unzip rtf-install-scripts.zip -d ./rtf-install-scripts ``` If this is for a PoC or demo purposes you can execute everything with root, but if this is for business usage then create a user and make it sudoer. Now that you’ve uploaded the installation scripts to every server, you are just a few steps away to install the Fabric. The installation is all based on scripts. The controller is the leader of all the work we are about to explain. I always like to understand first what I am going to execute, and then get into the box and do the work. Let’s do it. - We need to describe where we want to execute things and which role the different computes are going to take. And we will do it through setting a group of variables. Those variables will describe your intentions on deploying RTF. Some of those variables are optional. - Once those variables are set, one of the scripts that we have downloaded and then uploaded to our compute instances, is going to generate the config and the snippets to execute on every compute (controller, worker), depending on what you have defined on step 1. - The first snippet is executed on the controller. - Workers snippets can be executed in parallel, and will report themselves with the controller. Those are the high-level steps and a good way to understand what you are going to execute. Now let’s take a look to the variables we need to set: ![Table of RTF install variables with descriptions and the example values used here](../../assets/blog/mulesoft-runtime-fabric-deployed-on-oracle-cloud-infrastructure-oci-part-1-20.png) From all the information that is described in the previous table, up to this point you have all the information needed, but the `RTF_MULE_LICENSE`. The `RTF_MULE_LICENSE` is a file with extension `.lic` that you should have in case you have a valid and active subscription with MuleSoft. Once you have the file at your disposal, execute this: ```bash base64 -w0 license.lic ``` The output from that command is what you must set as the `RTF_MULE_LICENSE` variable value. The `RTF_ETCD_DEVICE` and `RTF_DOCKER_DEVICE` need to point to the volumes (storage) that we have provisioned in previous steps. ## Installation Execution Now it is time to execute the installation process. At this point we have some understanding of what we are about to execute and it is time to put all the pieces in place. 1. Go to the folder where you have unzipped the installer $HOME/rtf-install-scripts/ 2. Set the environment variables and generate the configuration ```bash RTF_CONTROLLER_IPS='11.0.0.5' \ RTF_WORKER_IPS='11.0.0.3 11.0.0.4' \ RTF_DOCKER_DEVICE='/dev/xvdc' \ RTF_ETCD_DEVICE='/dev/xvdb' \ RTF_ACTIVATION_DATA='f653432342-54433$4322' \ RTF_MULE_LICENSE='adfafasfasfasfasfasfdasdsff==' \ ./generate-configs.sh ``` 3. The execution of the previous step is going to generate and output like this: ```bash ========================================================= Runtime Fabric configuration generator ========================================================= Cluster topology: 1 controllers, 2 workers Instructions: 1. Create /opt/anypoint/runtimefabric directory and ensure it is writable from your ssh login 2. Copy each snippet below and execute on the appropriate machine 3. For each node, copy the init.sh script to the installation directory eg scp scripts/init.sh @node-ip:/opt/anypoint/runtimefabric 4. Execute sudo init.sh on each node. This should be done first on the leader (first controller) node, then concurrently on the other nodes 5. The nodes will join to the IP address given for the first controller and form your Runtime Fabric cluster. This process can take 10-25 minutes Note: You can monitor the progress of the installation on any of the nodes with tail -f /var/log/rtf-init.log 11.0.0.5: ========================================================= mkdir -p /opt/anypoint/runtimefabric && cat > /opt/anypoint/runtimefabric/env < /opt/anypoint/runtimefabric/env < /opt/anypoint/runtimefabric/env <>. Mule does not understand “\”. Make sure to replace “\” with “/”. So the final address you should be using for the “File Path” in the above configuration should be something like C:/Users/Documents/Test Files/<<File Name>>. Second Configuration You need to add the MIME Type. Since in this case we are dealing with an excel file, we will be keeping the MIME Type as application/xlsx. ![Read component MIME Type tab set to application/xlsx for the Excel file](../../assets/blog/dynamic-variable-in-mulesoft-6.png) **3.** Once this is done, we need to specify which sheet we are trying to read. A simple DataWeave script can help! ![Set Payload component with a DataWeave script reading payload."Sheet1"](../../assets/blog/dynamic-variable-in-mulesoft-7.png) By writing payload.“Sheet1”, we are instructing it to read the data which is there in the Sheet1. Now that you know how to read an excel file, we are good to go ahead. Let's explore now! ## Let's head back to our studio **1.** Once the file is read, we have logged the payload to see what is being fed to the mule flow (Refer to the prerequisites to learn about reading a file). ![Logged payload showing each record's Name, Dept, and Company as JSON objects](../../assets/blog/dynamic-variable-in-mulesoft-8.png) **2.** We can see that the payload is being read like above. So, to intercept the entire payload as individual chunks, we have used a for-each which divides the payload uniformly. **3.** On the onset of the flow, we have captured the queryString. A query string contains the total string which we have passed in the URL. ![GET request URL with the Company=ImaginaryA portion labeled as the query string](../../assets/blog/dynamic-variable-in-mulesoft-9.png) **4.** Once we reach inside the payload, we split the query string by the separator “=”. ![Set Variable named segBy with value vars.queryString splitBy "="](../../assets/blog/dynamic-variable-in-mulesoft-10.png) We can see **Company** as **segBy[0]** and **ImaginaryA** as **segBy[1]** (please check the above point for getting the references). **5.** Once that is done, we will write the DataWeave script to implement the dynamic variable concept. ![set-variable XML using a dynamicVar to build a variable name from segBy[0] at runtime](../../assets/blog/dynamic-variable-in-mulesoft-11.png) The name of the variable is kept in accordance with the feature, which we have mentioned in the query string, by which we want to segregate our records. In this case we want that feature to be Company. Here in the 3rd line, we have written dynamicVar = payload[vars.segBy[0] as String] as String - We have created a local variable named dynamicVar which will create a placeholder as per the number of companies in the payload. - This will be dynamic in nature, which means that the number of variables to be created will be decided during the runtime. The script that follows it simply checks if the company named variable is available or not. - If available, it pushes the records into that already available variable. - If not available, it creates the company named variable and then inserts the records. **6.** Once that is done, we will send an email to confirm whether our code has worked or not. In order to summon the variable, we will use the following script. ![Email Body config reading the dynamic variable via vars[vars.segBy[1]] as plain text](../../assets/blog/dynamic-variable-in-mulesoft-12.png) In our case since we have 4 companies in our payload, 4 variables will be created. Now, if you remember the query string, we want the records belonging to the company **ImaginaryA.** The command **segBy[1]** will return the value **ImaginaryA** (please check step 4). Hence, by the command we are trying to access the variable created for **ImaginaryA**. **7.** Once the process completes, as per the POC we should receive a mail which should look something like this: ![Received email listing only the ImaginaryA employee records in CSV format](../../assets/blog/dynamic-variable-in-mulesoft-13.png) So, finally we have a solution which can address our issue. ## GitHub repository [ProstDev GitHub - Dynamic Variable Mule](https://github.com/ProstDev/dynamic-variable-mule) --- ## My top 5 DataWeave tips to make your life easier Source: https://prostdev.com/post/my-top-5-dataweave-tips-to-make-your-life-easier | Published: Jun 1, 2021 | Category: Guides It took me some time to find and fully understand how to use these “hacks” *(they’re not really hacks, but they’ve made my life so much easier, so I’ll call them hacks)*. Trust me; they’re truly worth it, especially if you’re just starting to learn DataWeave. - Using the DataWeave Playground - Using the “dw” output - Using the “log” function - Using the “do” statement - Assigning types to variables and functions ## 1. Using the DataWeave Playground This is my absolute favorite tool, and I really hope more functionality gets added into it (*cough, cough, a DataWeave Notebook, please?).* I use this tool ALL the time. This is basically a tool that lets you experiment with DataWeave without doing it from Anypoint Studio. You can find the docker image to install in your local machine in this post (no previous Docker experience needed) - [How to run locally the DataWeave Playground Docker Image](https://www.prostdev.com/post/how-to-run-locally-the-dataweave-playground-docker-image)*.* But wait! Before you do that, the awesome DataWeave team already created an online site that you can access without having to run Docker. Check it out here: [https://developer.mulesoft.com/learn/dataweave](https://developer.mulesoft.com/learn/dataweave) ![DataWeave Playground web interface with input explorer, script editor, and JSON output panels](../../assets/blog/my-top-5-dataweave-tips-to-make-your-life-easier-1.png) ## 2. Using the “dw” output When you’re working with certain data types, like JSON, you don’t have the same visibility as some of the values’ types. For example, JSON converts the Dates into Strings. You can’t be absolutely sure about the exact type of output unless you follow the code. Here’s an example: ```dataweave %dw 2.0 output application/json --- "2000-01-01" as Date // "2000-01-01" ``` (I’m adding in the comments the output that I’m getting) Simply by looking at “2000-01-01”, you’re not sure if it’s a String or a Date (without looking at the code, of course). If I remove the “as Date” coercion, the output remains the same. ```dataweave %dw 2.0 output application/json --- "2000-01-01" // "2000-01-01" ``` Now try changing the “json” output for a “dw” output and see the magic. Without coercion: ```dataweave %dw 2.0 output application/dw --- "2000-01-01" // "2000-01-01" ``` With coercion: ```dataweave %dw 2.0 output application/dw --- "2000-01-01" as Date // |2000-01-01| ``` Ah! You can see the difference now! And that’s not all. By using this output, you can also see a function’s definition. Here’s an example of the “[plusplus](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-plusplus)” function: ```dataweave %dw 2.0 output application/dw --- ++ // (arg0:Array | String | Object | Date | LocalTime | Date | Time | Date | TimeZone | LocalDateTime | TimeZone | LocalTime | TimeZone, arg1:Array | String | Object | LocalTime | Date | Time | Date | TimeZone | Date | TimeZone | LocalDateTime | TimeZone | LocalTime) -> ??? ``` Pretty cool, right? ## 3. Using the “log” function Isn’t it frustrating when a small part of your script is not working, and you’re not sure why? You can’t actually debug it like you’d debug a Mule Flow. But guess what? You *can* output expressions into the console. Let’s say that I have the following script: ```dataweave %dw 2.0 output application/json --- (random() * 100) > 74 ``` I’m either getting true or false (Boolean, not String), depending on whether the randomly generated number is more than 74 or not, but I don’t know what the actual random number is. Well, I can use the “[log](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-log)” function to see which number it is. And the best part? Adding this into my script doesn’t affect anything I do. I don’t have to add a new variable only for the log. I can just surround the value that I want to log into the console in parentheses. ```dataweave %dw 2.0 output application/json --- log(random() * 100) > 74 ``` This is what it looks like from the DataWeave Playground: ![DataWeave Playground log viewer showing the random number 61.91 logged below the script](../../assets/blog/my-top-5-dataweave-tips-to-make-your-life-easier-2.png) You can experiment a bit with it to see what you can and can’t do with this function. For example, you can’t do this: ```dataweave %dw 2.0 output application/json --- log(if(true)) "ok" else "not ok" // Invalid input '"', expected Function Call ``` ## 4. Using the “do” statement You can create a local context by using the “[do](https://docs.mulesoft.com/mule-runtime/4.3/dataweave-flow-control#control_flow_do)” statement. This includes variables, functions, and the script. This way, you can have variables inside your functions that are not accessible outside of the function. I use this a lot to not end up with a lot of global variables that can’t be re-used for other purposes. Consider this script: ```dataweave %dw 2.0 output application/json var people = [ { name: "Alex", age: 90 }, { name: "Elsa", age: 24 } ] --- (people map do { var newName = if ($.name == "Elsa") "Anna" else $.name --- { name: newName, age: $.age } }) ``` ![DataWeave Playground running the given script and showing the transformed output.](../../assets/blog/my-top-5-dataweave-tips-to-make-your-life-easier-3.png) If we wanted to re-use the "newName" variable from outside of this context (set by "do"), we would receive an error because the variable only exists inside the context where it was created. ```dataweave (people map do { var newName = if ($.name == "Elsa") "Anna" else $.name --- { name: newName, age: $.age } }) + newName // Unable to resolve reference of: `newName`. ``` In this example, I created a context inside the “map” function, but notice how I need to re-declare the object for the map. This is because there’s a new script section inside the “map” (right under the three dashes (---). A regular map would look like this: ```dataweave people map { name: $.name, age: $.age } ``` I don’t need to add additional curly brackets ({ }) in the regular map. But because I opened a new context, I need to treat the new space just like a regular script. You *can* continue creating more contexts inside contexts, but please **don’t** do this. :D I’m not sure if this will create performance issues later, but it doesn’t look good, and you may end up with “[Spaghetti code](https://en.wikipedia.org/wiki/Spaghetti_code).” It took me some time and practice to get used to this statement. The best way to master it is to use it frequently over several days (not only a lot of times in one day but for many days). ## 5. Assigning types to variables and functions When I first started developing in DataWeave, I didn’t assign any [type](https://docs.mulesoft.com/mule-runtime/4.3/dataweave-type-system) to the functions or variables I was creating because it’s not really needed. Here’s an example: ```dataweave %dw 2.0 output application/json var str: String = "Hello" fun concat(str1: String, str2: String): String = str1 ++ str2 --- concat(str, " World!") // "Hello World!" ``` As you can see, my variable “str”, the two parameters of the “concat” function, and even the return type of the function itself, are all of type “String.” This has helped me in different ways: **1.** I don’t get confused when I have a big script and forget which kind of data a variable was. I can easily look at the data type I assigned to it. **2.** My mistakes and potential bugs are reduced because I know exactly what kind of data I *could* get back. For example, when you use a [selector](https://docs.mulesoft.com/mule-runtime/4.3/dataweave-selectors) like the Index selector, it *is* possible that the data you get is a Null, instead of the data you were expecting. ```dataweave %dw 2.0 output application/json fun getValue(array: Array, index: Number): Number | Null = array[index] // Return type can be a Number (because it's an array of numbers), but it can also be a Null if the index wasn't found. Hence the “Number | Null” union. --- getValue([1, 2, 3], 5) ``` **3.** I get early errors in my script before deploying and running my Mule App, saving me time. **4.** It’s easier for the next developer to understand what the code is doing by looking at the data types. **5.** Probably the best thing of all is that I learned how to use [Function Overloading](https://docs.mulesoft.com/mule-runtime/4.3/dataweave-functions#overloading-functions) when I’m expecting different functionality depending on the data type. ```dataweave %dw 2.0 output application/json fun concat(value1: String, value2: String): String = value1 ++ value2 fun concat(value1: Number, value2: Number): String = (value1 as String) ++ (value2 as String) fun concat(value1: Any, value2: Any): String = "Other types" --- { strings: concat("Hello", " World!"), // Hello World!" numbers: concat(20, 20), // "2020" other: concat({},[]) // "Other types" } ``` ## Recap - You can use the DataWeave Playground Docker Image locally by following [this post](https://www.prostdev.com/post/how-to-run-locally-the-dataweave-playground-docker-image), or directly using [the public site](https://developer.mulesoft.com/learn/dataweave) (if available). - Use the output “application/dw” to see the internal data structures or read the definition of a function. - You can “debug” your script by using the “[log](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-log)” function and output data into the console. - Create local variables and functions using the “[do](https://docs.mulesoft.com/mule-runtime/4.3/dataweave-flow-control#control_flow_do)” statement. - Assign [types](https://docs.mulesoft.com/mule-runtime/4.3/dataweave-type-system) to variables and functions to avoid confusion later, reduce mistakes, catch errors early, help the next developer understand your code, and use [Function Overloading](https://docs.mulesoft.com/mule-runtime/4.3/dataweave-functions#overloading-functions) when needed. I hope these tips help you in your DataWeave journey. *Prost!* -Alex --- ## Easiest way to integrate automatic code review of MuleSoft apps using SonarQube and Docker Container Source: https://prostdev.com/post/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container | Published: May 25, 2021 | Category: Tutorials ## Using SonarQube and Docker The quickest way to have an installation of SonarQube up and running is using a Docker container. SonarQube is a platform for continuous inspection of code quality to perform automatic reviews with static analysis of code to detect bugs, [code smells](https://en.wikipedia.org/wiki/Code_smell) and security vulnerabilities. This definition is not by me, it is from Wikipedia ([https://en.wikipedia.org/wiki/SonarQube](https://en.wikipedia.org/wiki/SonarQube)). If you want to know more about SonarQube and all its capabilities, you can visit their website: [https://www.sonarqube.org/](https://www.sonarqube.org/developer-edition/?gads_campaign=South-America-SonarQube&gads_ad_group=SonarQube&gads_keyword=sonarqube&gclid=Cj0KCQjwvYSEBhDjARIsAJMn0lga8FBAQxpXuiDOFvD4zK28tuA1_8TlLkBTOtDcFaWIahnsU2DtyKUaAgbGEALw_wcB) On the other hand, Docker delivers software in packages called containers (they are ready to use). It is very cool, isn’t it? If you want to learn more about Docker, please visit their website: [https://www.docker.com/](https://www.docker.com/) ## Docker Hub Let’s use Docker Hub (the world’s largest library and community for container images) to find a SonarQube Docker container to perform our MuleSoft applications code reviews. Step by step: - Go to Docker Hub: [https://hub.docker.com/](https://hub.docker.com/) - Sign in (or sign up if you do not have an account already) - Download the following image: mulesonarqube - Run the Docker image. That is all! Now you have a SonarQube instance up and running, and it is ready to perform MuleSoft code analysis. ## SonarQube Docker Container To download the Docker image (it will take a while): ```bash docker pull fperezpa/mulesonarqube:7.7.3 ``` ![Terminal showing docker pull of the mulesonarqube:7.7.3 image with all layers Pull complete](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-1.png) To run the Docker image: ```bash docker run -d --name sonarqube -p 9000:9000 -p 9092:9092 fperezpa/mulesonarqube:7.7.3 ``` You should see something like this in your command line: ![Terminal with docker run then docker ps showing the sonarqube container Up on ports 9000 and 9092](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-2.png) Or in your Docker Desktop: ![Docker Desktop showing the running sonarqube container's startup logs](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-3.png) If you want to know more about Docker Desktop (available for Windows, Mac and Linux), visit this website: [https://www.docker.com/products/docker-desktop](https://www.docker.com/products/docker-desktop) ## Go to your SonarQube instance That is it! Your SonarQube instance is ready to use. Go to: [http://localhost:9000/projects](http://localhost:9000/projects) (use admin/admin as default username and password to Log In, you can change it later). You should see something like this: ![Empty SonarQube Projects dashboard with a Create new project button](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-4.png) There is a set of Rules already configured in your SonarQube instance to analyze MuleSoft code. You can find those clicking on Rules (top menu) and then on Mule (left menu). If you are feeling lucky you can modify those rules as needed. ![SonarQube Rules page filtered by the Mule language, listing 54 Mule rules](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-5.png) There are two Quality Profiles: one for Mule 4 applications and another one for Mule 3 applications. The default Quality Profile is Mule 4. Explore it by clicking on Quality Profiles (top menu) and then filtering profiles by Mule (picking list). ![SonarQube Quality Profiles showing MuleSoft Rules for Mule 3.x and Mule 4.x, with 4.x default](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-6.png) Continue exploring your SonarQube, enjoy the trip! ## Important! XML configuration SonarQube already comes with an XML plugin. But…Your MuleSoft code is also XML. The plugin we are going to use inspects XML files, so you need to remove the XML files from the default XML plugin in SonarQube. This will avoid the XML plugin checks for your MuleSoft application and lets the Mule plugin into action. To do that, click on Administration (top menu), then Configuration and General Settings (top left menu), click on XML (left menu) and remove the .xml extension from the XML plugin configuration. You should see something like this (click on the red cross to remove .xml line): ![SonarQube XML plugin File suffixes setting listing .xml, .xsd and .xsl with remove crosses](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-7.png) Then click on the Save button and you are done! Your SonarQube is ready to check your MuleSoft code. ![XML File suffixes after removing .xml, now only .xsd and .xsl, with the Save button](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-8.png) ## Use the Maven Plugin We are about to use a Maven plugin to verify the quality of our MuleSoft code using SonarQube. This plugin contains a set of rules and metrics that are going to be used and calculated every time a project is being inspected. This is a plugin developed by MuleSoft and it is unlicensed. If you want to know more about this plugin, please refer to the official repository: [https://github.com/mulesoft-catalyst/mule-sonarqube-plugin](https://github.com/mulesoft-catalyst/mule-sonarqube-plugin) To use this plugin, open a command line and navigate to your MuleSoft project’s home directory and run the following command: ```bash mvn sonar:sonar -Dsonar.host.url=http://localhost:9000 -Dsonar.sources=.\src ``` - -Dsonar.host.url is the URL where your SonarQube instance is running. - -Dsonar.sources is the path where the code to analyze is (typically src folder under your MuleSoft project home). I have quickly created a Kafka producer-consumer MuleSoft app and I will analyze this code. You should see something like the following: ![Terminal output of mvn sonar:sonar starting analysis of the kafka-producer project](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-9.png) Finally, the plugin completes the code analysis, and the report will be available at your SonarQube instance: ![Maven output ending in ANALYSIS SUCCESSFUL and BUILD SUCCESS with the report URL](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-10.png) ## SonarQube Report You will find the report in your SonarQube instance. Just navigate to Projects (top menu), you should see your project there. ![SonarQube Projects list with the kafka-producer project Passed, showing 2 bugs and 5 code smells](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-11.png) Click on your project and explore the report details. ![kafka-producer project overview with Bugs, Vulnerabilities, Code Smells and Duplications metrics](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-12.png) Let’s take a quick look at the Bugs section. ![SonarQube Issues view listing two bugs including credentials should be managed with properties](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-13.png) ## Conclusion The report is highlighting a bug that says that credentials and resources should be managed with application properties. This was intentional. The application has an HTTP Listener with hardcoded host and port values. This is not a good practice to develop MuleSoft applications, those should be placed in a properties file. ![Bug detail showing the offending HTTP Listener config XML with hardcoded host and port 8081](../../assets/blog/easiest-way-to-integrate-automatic-code-review-of-mulesoft-apps-using-sonarqube-and-docker-container-14.png) You can also explore the Mule default Rules definition in your SonarQube instance and modify/add rules as needed. If you have different MuleSoft coding standards you can enrich that set of rules. This is the first step to semi-automate code reviews for MuleSoft projects. After this you can integrate this Maven plugin with a CI/CD pipeline. This is very cool, right? --- ## Trigger Events in Salesforce; Receive Them in MuleSoft Source: https://prostdev.com/post/trigger-events-in-salesforce-receive-them-in-mulesoft | Published: May 18, 2021 | Category: Tutorials We all know that MuleSoft is now part of Salesforce. It's been more than two years since Salesforce acquired it. But even before the acquisition, those two have had a strong integration. In my opinion, as a person who has been working in the integration space for more than 20 years, one of the most attractive things within their integration through the MuleSoft Salesforce connector, is the ability to subscribe to events that occur within Salesforce. One of the challenges that has always been present when you are integrating an application is the ability to generate events that 3rd parties can subscribe to. This is a good element for an Event-Driven Architecture (EDA), and really a differentiator if you are building a connectivity/integration initiative. Well, Salesforce offers a lot of alternatives to generate events. Not only within Salesforce boundaries, but also across them. And one key player here for that regard, is MuleSoft via its Salesforce Connector. Salesforce offers: - Push Topic Event - Generic Event - CDC - Platform Event - etc. Which are different alternatives to generate events within Salesforce, after a change has occurred in one of its objects or when you programmed that the event should occur (APEX, Workflow). Some of them are more flexible than others in terms of the customization and data that the event can contain. But something that they have in common, is that they can be used by the MuleSoft Salesforce Connector, and with that, trigger a MuleSoft flow. And now within a MuleSoft flow, you can pretty much do whatever you want with the data and send it to other systems, or publish the information through another event mechanism for 3rd parties. Let's imagine you want to inform another system every time a create, update or delete for a Lead happens. But you do not want to be polling Salesforce every now and then to look up for new Leads. What you want is to be notified that the event happened and then manipulate the information and send it to others, or process them, or whatever you want to do with it. In the last paragraph I said "you do not want to be polling," I mentioned it because that is a common practice, but a very bad one. Rather than doing that, you can subscribe for events and process them only when they happen and you are ready to receive them. That is a cleaner and more effective approach, and it can help you to understand the value of an Event-Driven Architecture. MuleSoft is a very versatile platform. It offers an API platform, but also an API implementation and integration platform. Not only that, it can be used to expose asynchronous APIs, to serve as part of an EDA architecture. In this post I am going to show you how we can create a MuleSoft Flow that can react to three different events generated by Salesforce: 1. On Create Event 2. On Modify Event 3. On Delete Event We will use Lead as the entity in Salesforce. We are interested to know if a Lead was created, updated, or deleted. But again, we want to react to the event. We are not going to be asking Salesforce if a Lead was created, updated or deleted. No! We are going to subscribe and react. ## Creating the Mule Project In order to do that, let's create a MuleSoft project that will look like this: ![Mule project with three flows, each a Salesforce trigger (New, Modified, Deleted Object) into a Logger.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-1.jpg) We have three different flows: one per event. After we receive the event, we simply send it to the logger. In order to achieve this, we need two things: - Create the MuleSoft project and use the Salesforce connector. In order to configure the Salesforce connector, you need a developer account ([https://developer.salesforce.com/](https://developer.salesforce.com/)). - Create a Lead in Salesforce and play with it: update it and delete it. The steps are: 1) Create a MuleSoft Project: ![New Mule Project dialog with Mule Server 4.3.0 EE runtime selected and an empty project name.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-2.jpg) 2) Name it: “triggers01” or any name you would like to use. 3) Once the project is created, let's create a properties file that will include the Salesforce connectivity information. ![salesforce.yaml under src/main/resources holding sfdc username, password and token properties.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-3.png) The token is generated via the developer.salesforce.com portal, as well as the user and password. 4) The properties (yaml) file needs to be located in the folder that is indicated in the previous image (`src/main/resources`). 5) Now let's create two entries in our Global Elements tab: ![Global Configuration Elements tab with Configuration properties and Salesforce_Config entries.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-4.jpg) 6) For the Salesforce Configuration, you should input this: ![Salesforce Config properties with Basic Authentication username, password and security token from the YAML.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-5.jpg) As you can see we have configured username, password and token. We’re getting the values from the properties file. It is a good practice to use this type of configuration. Once you typed it, click on Test Connection: ![Salesforce Config showing a "Test connection successful" confirmation dialog.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-6.jpg) 7) Now it’s time to create the flow. As we've previously mentioned, we will react to events. Therefore, we are not going to drag and drop HTTP listeners. We are going to drag and drop a Salesforce Connector capability. 8) Let's start with the On Create Event. But before that, make sure you have added the Salesforce Connector into your Mule Palette. Like this: ![Mule Palette with the Salesforce connector module added alongside Core and HTTP.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-7.jpg) 9) Now, let's add the On New Object. Please locate it, and drag and drop it in the Canvas: ![Salesforce palette operations list with the On New Object trigger highlighted.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-8.jpg) 10) Now you should have this: ![A single flow with the On New Object Salesforce trigger dropped onto the canvas.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-9.jpg) 11) Repeat the same formula for these two events: ![Salesforce palette with the On Modified Object and On New Object triggers highlighted.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-10.jpg) 12) Drag and drop them into the canvas. And now you should have this: ![Three flows each starting with a Salesforce trigger: On New, On Modified, and On Deleted Object.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-11.png) 13) Now let's add a logger activity into the On New Object flow, and configure it like this: ![Logger added after the On New Object trigger, configured with the payload as its message.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-12.jpg) 14) Repeat the same formula for the other two flows. Now you will have this: ![All three flows now complete: each Salesforce trigger connected to its own Logger.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-13.jpg) 15) Now let's configure the three events. Let's start with the On New Object event. Double click the activity and configure it like this: ![On New Object config: Lead object type, Fixed Frequency scheduling, every 5 seconds.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-14.jpg) As you can see we've configured different elements: - Display Name: It can be whatever you want. - Connector Configuration: It should be already selected, since you created it in previous steps. - Object Type: This is the entity to which you want to be informed if a new one is created, modified or deleted. In this case we are going to use Lead. - Scheduling Strategy: It is the frequency to which we are going to get the event messages. - Since: This is a key element because events in Salesforce are independent of the Connector. So, if you are already publishing events when you start the MuleSoft app, it will retrieve the messages that were created before. But if you choose a date, then only the events created after that date will be retrieved. So be careful with this field. - Frequency: It is related to the Scheduling Strategy. In our case it will be every five seconds. - Time Unit: In this case, seconds. 16) Now let's do the same thing for the On Modify and On Delete, like this: ![On Modified Object config with object type Prospecto (Lead) and Fixed Frequency scheduling.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-15.jpg) and ![On Deleted Object config with object type Prospecto (Lead) and Fixed Frequency scheduling.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-16.jpg) 17) Now we are ready to test this. Right click your project and Run the project. The project should be in the DEPLOYED status. Like this: ![Mule console showing the triggers01 application in DEPLOYED status on Mule Server 4.3.0 EE.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-17.jpg) Now we are ready to create some Leads in Salesforce, and see how our MuleSoft application reacts to them. ## Triggering the Mule Flows Go to https://salesforce.com and log in with your credentials. Look for the Leads tab: ![Salesforce Lightning Sales home in Spanish with the Prospectos (Leads) tab in the navigation.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-18.jpg) (Here you see it in Spanish, because my defaults are in Spanish. But in English it is “Lead”). Let's create a new Lead: ![New Lead form in Salesforce filled in for Jose Perez at company SPS, with email perez@me.com.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-19.jpg) Just fill the necessary elements, and click create the Lead. After that, get back to Anypoint Studio, and see what was written in the log: ![Two Mule log lines from triggers01Flow and triggers01Flow1 logged after creating the Lead.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-20.jpg) You have two new lines, and if you scroll to the right, you will see the details of the Lead you've created: ![Logged Lead details scrolled right, showing FirstName=Jose and Title=CTO for the new record.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-21.jpg) Why two? Every time you create an object, it generates two events: one for creation, and one for modification. Now let's test the modification event. Let's edit the Lead changing the contact email: ![Lead detail page for Mr. Jose Perez with the email field perez@me.com highlighted before editing.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-22.jpg) Let's edit the email address: ![Editing the Lead's email field to jose_perez@me.com in the Salesforce inline editor.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-23.jpg) After you've edited, you will see a 3rd line in your log, where you can see the updated email address: ![Mule log of the modify event showing the updated Email=jose_perez@me.com value.](../../assets/blog/trigger-events-in-salesforce-receive-them-in-mulesoft-24.jpg) Pretty simple, right? With this, you can change the perspective on how you can integrate Salesforce with other applications. The connector is very powerful and the events layer in Salesforce as well. And they are integrated, so you can do the math and conclude that this is a great ability within MuleSoft. --- ## Mule 4 Unsplash Connector Source: https://prostdev.com/post/mule-4-unsplash-connector | Published: May 11, 2021 | Category: Tutorials *GitHub repositories with the projects can be found at the end of the post.* ## What is Unsplash? Unsplash is a website dedicated to sharing stock photography under the Unsplash license. Since 2021, it has been owned by Getty Images. The website claims over 207,000 contributing photographers and generates more than 17 billion photo impressions per month on their growing library of over 2 million photos. Basically Unsplash allows you to download images for free and use them under Unsplash license on your web pages and projects. Personally I’m a big fan of this community. I’ve been uploading photographs for the last couple years and it’s really amazing how many of the images uploaded have been used in a number of sites like BuzzFeed, Adobe Spark and Trello. Also many people share on Twitter and email how they wanna use my images. Unsplash has a [REST API](https://unsplash.com/documentation) that provides a bunch of nice features like image search, photo actions, collections, topics and users. Basically you can use all these actions to create any kind of automated application. This time I have created a new custom connector for Mule 4 to use a couple operations (Get Random Photo and Search Photos). ## Creating the custom connector Creating a custom module (connector) is easier than you think, so let me list the steps I followed. 1. Executing the maven archetype command generate in your terminal. ```bash mvn org.mule.extensions:mule-extensions-archetype-maven-plugin:generate ``` 2. Then a few questions need to be answered related to the project. ``` Enter the name of the extension: Enter the extension's groupId: Enter the extension's artifactId: Enter the extension's version: 1.0.0 Enter the extension's main package: ``` 3. Run the mvn eclipse:eclipse command in the terminal, so this way when we can import our projects and let it be recognized by the IDE (in case you are using Eclipse IDE). I personally like IntelliJ IDEA. 4. Import the generated project into the IDE. In my case, since I’m using the IntelliJ IDEA I select the maven project (*it’s easier if the IDE is in charge of pulling the maven dependencies into the project*). ![IntelliJ IDEA Import Project dialog with the Eclipse external-model option selected](../../assets/blog/mule-4-unsplash-connector-1.png) Finally, the connector is imported and we are able to start working on it. 5. Now for this specific connector we are going to use only two of the auto-generated java classes: **UnsplashConfiguration** and **UnsplashOperations.** - **UnsplashConfiguration.** Allows to set up any general parameter we might use. In this case we needed to set the Unsplash Access Key required for API calls. - **UnsplashOperations**. Here we will specify our first operations in the connector. We should be able to include more operations later. For now we will only have **GetRandomPhoto** and **SearchPhoto.** - **GetRandomPhoto.** Will retrieve a random picture from the API. - **SearchPhoto.** Will let you search based on a keyword and retrieve some picture information. ## Get Random Photo This operation will allow you to get a random set of photos from Unsplash and be able to specify some additional attributes like count (number of images you want to retrieve), keyword (specify the keyword for images you want to get from Unsplash) and orientation (it can be portrait, landscape or squarish). The connector will show the operations this way: ![Get random photo operation config with Count, Keyword, and Orientation portrait/landscape/squarish](../../assets/blog/mule-4-unsplash-connector-2.png) ## Search Photo The search operation allows you to search photos on the Unsplash API specifying a few optional attributes like **search terms** (this will tell the search the photos you are looking for), **page** (specify the page you are in), and **per page (**tells how many records per page you want). ![Search photo operation config with Search term, Page, and Per page fields in Anypoint Studio](../../assets/blog/mule-4-unsplash-connector-3.png) ## Installing the Mule module The installation is really easy. Basically we just need to run the command: ```bash mvn clean install ``` After running the command from terminal or IntelliJ IDEA, the new module will appear on the add modules palette. ![Mule Palette Add Modules view with the custom Unsplash module listed under Workspace](../../assets/blog/mule-4-unsplash-connector-4.png) Then we just need to drag and drop the module for our project. In this specific case I have created a simple application that uses both operations. Search operation in this case contains a foreach component that allows you to get the photo from Unsplash API and place the actual image in your local directory. Here is a sample of images downloaded from Search: ![Two Mule flows using the Unsplash operations and a folder of photos downloaded from Search](../../assets/blog/mule-4-unsplash-connector-5.png) ## GitHub repository You can get both projects from the repositories: Connector: [https://github.com/emoran/mule4-unsplash-connector](https://github.com/emoran/mule4-unsplash-connector) Connector implemented code: [https://github.com/emoran/unsplash-connector-implementation](https://github.com/emoran/unsplash-connector-implementation) Hope you enjoy this project and hopefully it will help with any type of project / integration you’re working on. --- ## ProstDev's New Blog Post Template Source: https://prostdev.com/post/prostdev-s-new-blog-post-template | Published: May 4, 2021 | Category: Guides Hi there! You’re probably here because you want to create a blog post with us. Awesome! Let’s get started. If you’re a new writer, please take a moment to read through this post to understand the different guidelines and standards when publishing in ProstDev. Also, feel free to look at the other content creation articles to read about best practices and tips for writing: [Content Creation articles](http://www.prostdev.com/blog/categories/content-creation). - Google Doc Template - Activating Word Count - Blog Post Title (under 100 characters) - Excerpt (max. 140 characters) - SEO Description (max. 500 characters) - Article Formatting - Start the Review Process ## Prerequisites If you’re not familiar with any of the following tools/terms, please take a moment to get acquainted with them before starting this guide. - [Google Drive / Google Docs](https://support.google.com/drive/answer/2424384?co=GENIE.Platform%3DDesktop&hl=en) - [SEO (Search Engine Optimization)](https://en.wikipedia.org/wiki/Search_engine_optimization) - [Gist files](https://docs.github.com/en/github/writing-on-github/creating-gists) ## 1. Google Doc Template First of all, open the following Google Doc template to have a starting point: [https://docs.google.com/document/d/1oCEdZXUnRY86MOogWeTR6Jm5IbGyvVhqoyjD9oXY-eA/edit?usp=sharing](https://docs.google.com/document/d/1oCEdZXUnRY86MOogWeTR6Jm5IbGyvVhqoyjD9oXY-eA/edit?usp=sharing) Make sure to be logged in to Google Drive, or click on the “Sign In” button in the top right corner. ![Top right corner when you’re not signed in.](../../assets/blog/prostdev-s-new-blog-post-template-1.png) ![Top right corner after you’re signed in.](../../assets/blog/prostdev-s-new-blog-post-template-2.png) After this, you can select File > Make a copy. ![Google Docs File menu open with the "Make a copy" option highlighted](../../assets/blog/prostdev-s-new-blog-post-template-3.png) Change the name and select in which folder you want to save it. Click “OK.” ![Google Docs "Copy document" dialog with a name field and a folder set to My Drive](../../assets/blog/prostdev-s-new-blog-post-template-4.png) Now you can start editing the template with your own content! > [!NOTE] > Please continue using Google Docs to create your post because the review process will happen on this platform. You can also create your post in other platforms or tools, but you’ll then have to move the content into a Google Doc to continue with the process. ## 2. Activating Word Count This is a neat trick that I *always* use when I’m reviewing or writing a post. In your Google Doc, select Tools > Word count. ![Google Docs Tools menu open with the "Word count" option highlighted](../../assets/blog/prostdev-s-new-blog-post-template-5.png) After this, activate the “Display word count while typing” option and click “OK.” ![Word count dialog with the "Display word count while typing" checkbox enabled](../../assets/blog/prostdev-s-new-blog-post-template-6.png) This will add a floating menu/button in the bottom left corner of the document to see the different counts. ![Floating word-count button in the document showing page, word, and character totals](../../assets/blog/prostdev-s-new-blog-post-template-7.png) If you haven’t selected any text, the count will be from the entire document. If you highlight some text, then this count will be only from the selected area. This will help you to keep the desired word/character count in the next steps. > [!NOTE] > Try to keep your article less than 2,000 words if possible. The average is 1,500. If you’re writing a very long post, it’s best to break it down into several articles and create a series instead. ## 3. Blog Post Title (under 100 characters) You can find some ideas to choose your title in this post - [7 tips to start writing your technical blog post](https://www.prostdev.com/post/7-tips-to-start-writing-your-technical-blog-post)*.* You will want to include some keywords so people can find your content more easily when they search on Google or other search engines (but keep the title under 100 characters!). For example, if your content is about programming in Java, make sure to include the keyword “Java” in your title. Or if the article is also focusing on showing how the “toString” method works, then have that keyword too. A good title could be something like “How to use the toString method in Java.” Identify between 2-5 keywords that *must* be included in the title first, and then see how you can structure it to be under 100 characters. If your title is too long, some keywords will have to be cut out, but don’t worry! You’ll learn how to still include them in other parts of the post for a better [SEO](https://en.wikipedia.org/wiki/Search_engine_optimization). ## 4. Excerpt (max. 140 characters) Add a very short description (less than 140 characters) of your content so the audience can get a better idea of what they’re about to read. This can also be a “hook” to keep readers interested in your content. > [!NOTE] > It’s not necessary to include any keywords here. This is just for the actual reader and not for the search engines. For example, “In this post, I explain how to use the ‘toString’ method and some of the most performant ways to use it.” ## 5. SEO Description (max. 500 characters) *Now* we’re back to the SEO keywords that you couldn’t include in your title from step 3. Add a paragraph of 500 characters or less with the keywords that didn’t fit in the title because of the character restriction. This text does not have to appear in the actual article. It can be added to the SEO Description without the need to add it to the post. This description will appear in the search results (for example, from Google), so try to think like your target audience. Ask yourself, “What would a person be *googling* to find my article?” For example, “How to use the ‘toString’ method in Java. Understanding ‘toString’. Transforming an Integer / Boolean into a String.” Try searching for some stuff yourself to see what the description includes in the results to give you a better idea of what you could put in this paragraph. Just make sure that what you add in the SEO Description is something you actually included in your content. Otherwise, your audience will feel “fooled” and won’t come back to see more *(also, it’s not cool to add keywords just to get clicks. You’re creating quality content!)*. ![Example of a Google search with the keywords "dataweave concatenate strings"](../../assets/blog/prostdev-s-new-blog-post-template-8.png) ## 6. Article Formatting Add the first subtitle after the introduction (steps 3-5) to separate the sections. After this, you can start developing your article. Some formatting to keep in mind: **1. Screenshots/pictures** If you’re adding a screenshot or a picture, make sure you explain what is demonstrated before you show the image. You can also include additional information about the image under the picture. For example: ![Screenshot taken from the "Anypoint Platform Chrome Extension" blog post by Edgar (Yucel) Moran.](../../assets/blog/prostdev-s-new-blog-post-template-9.png) **2. Code** Same as an image, explain what the code is doing before you show it. There are 3 different types of code that can be added in the article: in-line code, small scripts (approx. 1-10 lines), or large scripts (more than 10-15 lines). Unfortunately, there’s no formatting for in-line code, so please add your variable name, property name, or any other code in between quotes. For example: *Use the “secondVar” variable to...* Short pieces of code can be directly included in the post. For example: ![Screenshot taken from the "How to integrate Solace PubSub+ Cloud with MuleSoft" blog post by Pravallika Nagaraja.](../../assets/blog/prostdev-s-new-blog-post-template-10.png) Large pieces of code should be added in a [gist file](https://docs.github.com/en/github/writing-on-github/creating-gists), and then provide the gist file link in your Google Doc. For example: [https://gist.github.com/alexandramartinez/798a12afe843c7218ae2564b9a0bc9d7](https://gist.github.com/alexandramartinez/798a12afe843c7218ae2564b9a0bc9d7) The final output will look something like this: ![Screenshot taken from the "How to integrate Solace PubSub+ Cloud with MuleSoft" blog post by Pravallika Nagaraja.](../../assets/blog/prostdev-s-new-blog-post-template-11.png) ## 7. Start the Review Process Once your article is all done, click that “Share” button in the top right corner of the screen. ![Top-right corner of Google Docs with the blue "Share" button highlighted](../../assets/blog/prostdev-s-new-blog-post-template-12.png) Select the “Anyone with the link” option, and you can either select “Commenter” or “Editor” as the role. The difference is that a “Commenter” can *suggest* changes, but you have to review and accept them, whereas an “Editor” can modify the document just as you can. If you’re not familiar with the review process, you can select “Editor.” ![Google Docs share dialog set to "Anyone with the link" with the role dropdown showing Editor](../../assets/blog/prostdev-s-new-blog-post-template-13.png) After that, click the “Copy link” button. Now go to [ProstDev.com/contact](https://www.prostdev.com/contact) and go to the “Contribute” section. Paste the copied link into the “Link to your article” text box. Fill out the rest of the fields, and click “Submit.” ![Contribute form found in ProstDev.com/contact to submit your content.](../../assets/blog/prostdev-s-new-blog-post-template-14.png) That’s it! After that, we’ll review your article and provide feedback if anything needs to be changed. Once the review process is complete, we’ll schedule the content and create the social media. ## Recap - **Make a copy** of the Google Doc Template into your own Google Drive. ([https://docs.google.com/document/d/1oCEdZXUnRY86MOogWeTR6Jm5IbGyvVhqoyjD9oXY-eA/edit?usp=sharing](https://docs.google.com/document/d/1oCEdZXUnRY86MOogWeTR6Jm5IbGyvVhqoyjD9oXY-eA/edit?usp=sharing)). - **Activate the Word Count** option and try to keep your document less than 1,500-2,000 words. - **Create your title** using keywords for a better SEO (max. 100 characters). - **Create the post’s excerpt** for the reader (max. 140 characters). - **Create the post’s SEO Description** (not added to the blog post) with additional keywords for the search engines (max. 500 characters). - **Make sure to add explanations** before adding media and follow the code formatting for in-line, short, or large scripts. - **Share your document** at [www.prostdev.com/contact](https://www.prostdev.com/contact) to start the review process. If you have additional questions, take a look at the other blog posts in the [Content Creation category](https://www.prostdev.com/blog/categories/content-creation) for best practices and advice. You can also contact us or comment here if you still have questions. Hope to see your articles soon :D *Prost!* -Alex --- ## The Power of cURL - Part II Source: https://prostdev.com/post/the-power-of-curl-part-ii | Published: Apr 27, 2021 | Category: Tutorials Welcome to part II of the The Power of cURL. If you are new to cURL, I recommend checking out my previous post [here](https://www.prostdev.com/post/the-power-of-curl). There I provide an overview of cURL and how to invoke HTTP methods from the command line. I go into detail on how to invoke APIs using HTTP GET and POST requests, as well as how to pass query parameters, URI parameters and headers. This post will explain how to invoke APIs using HTTPS via **cURL**. ## Review of cURL [cURL](https://curl.se/) is a tool used to transfer files. The cURL command can be used inside scripts or from the command line. cURL provides support for common protocols like HTTP, HTTPS, FTP and much more. This article will focus on using the cURL command to invoke integrations that use HTTPS. ## cURL Requests cURL commands begin with the keyword “curl” followed by an option. REQUEST OPTION: To specify the request method in an HTTP/HTTPS request. Use the --request or -X options. - SYNTAX: ```bash curl --request '' ``` - OR: ```bash curl -X '' ``` > [!NOTE] > This article does not use ' ' around the url but I recommend using quotes when passing query parameters. If possible, try to make using single-quotes around the url a habit. ## Demo Time The cURL commands provided in this article will be demonstrated by first invoking the JSON Placeholder API. It is a fake API used for building prototypes. Learn more about it [here](https://jsonplaceholder.typicode.com/). The latter portion of the article will be demonstrated by invoking a sample Mule API configured for HTTPS. Let’s get started! Let's make a request to the infamous JSON Placeholder API. EXAMPLE REQUEST ```bash curl --request GET https://jsonplaceholder.typicode.com/todos ``` ![Terminal showing a curl GET to the JSON Placeholder API returning a list of to-do objects](../../assets/blog/the-power-of-curl-part-ii-1.png) Notice we get a JSON list of to-do objects. **Simple, right?** This works because the server hosting [https://jsonplaceholder.typicode.com](https://jsonplaceholder.typicode.com/) has a digital certificate that is issued by a trusted Certificate Authority (CA). When an HTTPS request is invoked, the server will present its certificate. If the certificate is in the API client’s trust store, the API client recognizes the server as trusted and will accept the API’s response. **What happens if the server hosting the API is not trusted? Or not in the API client’s trust store?** This time I have configured HTTPS via TLS on a simple Mule API on my local machine. When invoking the API through curl, I get the following error. For this example, I have used a self-signed certificate. EXAMPLE REQUEST ```bash curl --request GET https://localhost:8081/curl/helloget ``` ![Terminal showing the curl error "SSL certificate problem: self signed certificate"](../../assets/blog/the-power-of-curl-part-ii-2.png) Notice the error I get back when invoking the API. “curl: (60) SSL certificate problem: self signed certificate.” This error occurred because the certificate was not from a trusted source. This certificate is not present in machines. But why is cURL screaming? The self-signed certificate is not trusted, that means it is not in my computer’s trust store. cURL offers 4 options in which one can resolve this type of error. I will go over 2 options but feel free to check out all four options [here](https://curl.se/docs/sslcerts.html). **Option 1:** Disable SSL certificate validation* A work-around to this is to add the --insecure or the -k option. These two options disable peer SSL certificate validation. - SYNTAX: ```bash curl --request '' --insecure ``` - OR: ```bash curl --request '' -k ``` > [!NOTE] > This option is insecure and disables SSL certificate validation. I recommend only using this option when invoking and/or testing internal APIs and you are certain the host is from a trusted source. EXAMPLE REQUEST ```bash curl --request GET https://localhost:8081/curl/helloget --insecure ``` ![Terminal showing a successful Mule API response after adding the curl --insecure option](../../assets/blog/the-power-of-curl-part-ii-3.png) **Option 2:** Provide the certificate during the invocation This option requires you to retrieve and save the certificate (or public key) of the API host. Then save this certificate in a location on your machine and reference it during the API invocation. Unlike option 1, SSL certificate validation is still in place and is therefore a safer option. - SYNTAX ```bash curl --request --cacert '' ``` EXAMPLE REQUEST ```bash curl --request --cacert C:Desktop/demo.cer https://localhost:8081/curl/helloget ``` ![Terminal showing a successful Mule API response using the curl --cacert certificate option](../../assets/blog/the-power-of-curl-part-ii-4.png) Notice in both option 1 and option 2, we were able to get a response from the Mule API. ## Conclusion cURL is a powerful tool. The 2 articles in this series only scratch the surface of cURL. If you would like to learn more, review the [cURL man page](https://curl.se/docs/manpage.html) and the [cURL ebook](https://everything.curl.dev/). Lastly, I am Whitney Akinola. I am a fellow MuleSoft Developer and content creator. Feel free to check out my technical content[here](https://www.whitneyakinola.io/blog). I am also looking for integration topics to write about. If you have a topic you would like me to explore and write about, please feel free to request. You can [contact me](https://linktr.ee/whitneyakinola) personally via LinkedIn or Twitter. --- ## 3 ways to import a function/module in DataWeave 2.0 Source: https://prostdev.com/post/3-ways-to-import-a-function-module-in-dataweave-2-0 | Published: Apr 20, 2021 | Category: Guides There are different ways to import a module in DataWeave. You need to do this for the modules that are not a part of the [core functions](https://docs.mulesoft.com/mule-runtime/4.3/dw-core). Here is a list of all the modules in DataWeave: [Mule Docs - DataWeave Reference](https://docs.mulesoft.com/mule-runtime/4.3/dw-functions). ## 1. Import module directly You can simply import the module without specifying the functions’ names. If you do this, you will need to reference the module every time you want to use that module’s function. Like this: ```dataweave %dw 2.0 output application/json import dw::core::Strings --- Strings::capitalize("alex martinez") ``` To reference the function, you need to use the name of the module, then add two colons (::), and finally, the name of the function. After doing this, you can continue using the function as usual. ## 2. Import all functions from a module You can import all of the functions from the module by using the asterisk (*) and using the keyword “from.” This will let you use the function directly without referencing the module’s name every time. Like this: ```dataweave %dw 2.0 output application/json import * from dw::core::Arrays --- { field1: [true, false] every ($), field2: [true, false] some ($) } ``` This approach, however, will import all of the functions that are found inside the module. Normally, this doesn’t affect performance. But I personally want to see which functions exactly I am importing to my code. For example, in the previous script, I don’t know if “every” or “some” are functions that come from the Arrays module or if they’re core functions. For this reason, my favorite approach is the 3rd option described below. ## 3. Import one or more functions from a module To explicitly import each function separately, you have to list them instead of just writing the asterisk (*). Like this: ```dataweave %dw 2.0 output application/json import every, some from dw::core::Arrays --- { field1: [true, false] every ($), field2: [true, false] some ($) } ``` This will still let you call the function directly. The difference is that you’ll have to list all of the functions you’re using. I prefer this approach because sometimes you may not know which function comes from which module (especially for big scripts). With this, you can just search for the function and see which module is importing it. Of course, it can become annoying if you’re changing functions often, and you remove and re-add them several times. You will also get an error if you forget to list one of the functions that you’re using. For example, the following script shows an error because the “some” function is not listed in the import statement: ```dataweave %dw 2.0 output application/json import every from dw::core::Arrays --- { field1: [true, false] every ($), field2: [true, false] some ($) // Unable to resolve reference of: `some`. } ``` The option you choose really depends on your personal preference or the standards of your project. By the way, do you know how to create your own modules and reference them for your DataWeave scripts? If not, check this out! - [Custom Modules in Mule 4 - DataWeave 2.0](https://www.prostdev.com/post/custom-modules-in-mule-4-dataweave-2-0). -Alex --- ## How to integrate Solace PubSub+ Cloud with MuleSoft - Negative Scenario (Error Handling) Source: https://prostdev.com/post/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling | Published: Apr 13, 2021 | Category: Tutorials *GitHub repository with the Mule project can be found at the end of the post.* In my previous post ([How to integrate Solace PubSub+ Cloud with MuleSoft](https://www.prostdev.com/post/how-to-integrate-solace-pubsub-cloud-with-mulesoft)), I explained the happy path of the Solace PubSub+ and MuleSoft Integration. In this post, let us see how we can handle an error efficiently. Even though the error handling logic varies depending on use cases, business requirements, organizational processes, etc. I believe the core approach to error handling for message-based integrations would more or less remain the same for different applications. The most common approach consists of the following steps: - In case of an error, the error is captured in the error handler logic. - Enrich the message with additional information that could be later used for reprocessing. - Keep track of the “deliveryCount” and other useful information in the “User Properties.” - Publish the message to an Error Topic for retrying. - Process the message from the error queue and depending on the “redelivery_count” attribute, either route it back to the source queue or the dead letter queue (or Dead Message Queue). Further reprocessing can happen by the operations team. Let us see how I have implemented the above approach in the following use case. Before that, please take a look at my [previous blog post](https://www.prostdev.com/post/how-to-integrate-solace-pubsub-cloud-with-mulesoft) on how to publish the message to the Solace Topic using the JMS “Publish” connector. **1)** The message is received by the “On New Message” JMS Connector. The incoming message is transformed in the “Request Transformation” component, While attempting to create the Salesforce Account using the “Create” connector, there is an error (in case the account already exists). The choice router validates if a successful response is received from SFDC. In case of failure, an error is thrown. (Note that you can also add an “Until Successful” block around the “Create” SFDC connector to retry for a specified amount of time before throwing an error.) ![Subscribe flow receiving a JMS message, transforming it, and creating a Salesforce Account with a Choice router](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-1.png) The error is caught in the “On Error Propagate” error handler and invokes the “processSFDCError” flow. ![Choice router branches plus an On Error Propagate handler referencing the processSFDCError flow](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-2.png) ![processSFDCError flow: Transform Message, Logger, and a sendNegativeAck Flow Reference](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-3.png) In case of Salesforce Connectivity error, then the error is thrown in the “Create” SFDC connector and caught in the “SFDC:ERROR” (On Error Propagate) as the error mapping is done in the connector level as shown in the following screenshot. ![Salesforce Create connector Error Mapping mapping ANY error type to SFDC:ERROR](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-4.png) **2)** The message is published to the Error Topic as shown in the following screenshot: ![sendNegativeAck flow setting error variables then publishing the message to the error queue](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-5.png) Note that I am capturing the “DELIVERYCOUNT”, “EXCEPTIONDETAILS” and “TIMESTAMP” in the “User Properties” field. ![JMS Publish config to the error topic with DELIVERYCOUNT, EXCEPTIONDETAILS, and TIMESTAMP user properties](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-6.png) **3)** The message from the Error Queue is received in the retry-flow. In this flow, if the “redelivery_count” (`attributes.properties.jmsxProperties.jmsxDeliveryCount`) value is less than or equal to the “maxRedeliveryCount” variable, which will be fetched from the properties file, then the message will be published to the source Topic. ![retry-flow listening on the error destination and setting flow, destinationTopic, and deadLetterTopic variables](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-7.png) ![Solace sample-test-error-queue showing one queued message after the first failure](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-8.png) Since the redelivery count is 1, the message is published back to the source Topic. ![Mule Debugger on the retry-processing-flow Choice that checks redelivery_count against maxRedeliveryCount](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-9.png) ![Solace sample-test-queue showing the message republished to the source queue](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-10.png) **4)** The message is received by the subscribing flow and tries to process it. In case of error during the second time, the message is again published to the Error Topic with redelivery_count=2. ![Mule Debugger variables showing redelivery_count = 2 after the second failure](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-11.png) **5)** In case of failure during the third retry, the message is again published to the Error Topic with redelivery_count=3. ![Mule Debugger variables showing redelivery_count = 3 after the third failure](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-12.png) **6)** Since the “redelivery_count” now equals the “maxRedeliveryCount” variable, the message is published to the Dead Letter Topic or Dead Message Topic. A notification email can be triggered to the operations team for further processing of this message. ![Debugger showing redelivery_count equals maxRedeliveryCount 3, so the message is published to the DLQ](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-13.png) ![Solace sample-test-dead-letter-queue showing the message after the max retries were exceeded](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-14.png) ![Solace Queues list with the dead-letter, error, and source test queues and their message counts](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-15.png) The message in the Error Topic is Acknowledged with the “retryAckId” variable as shown in the following screenshot. ![retry-flow acknowledging the error-topic message with the retryAckId variable](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-16.png) **7)** In case of a Transformation error, the message can be sent directly to the Dead Letter Queue as retrying does not make any sense. ![process-transformation-error flow publishing the message straight to the DLQ instead of retrying](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-negative-scenario-error-handling-17.png) Thank you for reading my blog post :) Happy Learning, Pravallika Nagaraja ## GitHub repository [ProstDev GitHub - Solace MuleSoft Integration](https://github.com/ProstDev/solace-mulesoft-integration) --- ## How to integrate Solace PubSub+ Cloud with MuleSoft Source: https://prostdev.com/post/how-to-integrate-solace-pubsub-cloud-with-mulesoft | Published: Apr 6, 2021 | Category: Tutorials *GitHub repository with the Mule project can be found at the end of the post.* In this article, I will demonstrate how to integrate Solace PubSub+ with MuleSoft to publish and consume the messages. ## What is PubSub+? PubSub+ is a platform that enables the design, discovery, streaming and full lifecycle management of events across distributed enterprises. It provides all of the tools necessary to support a modern event-driven architecture. The main components of the platform are PubSub+ Event Brokers, PubSub+ Event Portal and PubSub+ Cloud Console. *More information in the* [PubSub+ for Developers](https://www.solace.dev/) *website.* ## Advantages and Disadvantages of Event-Driven Approach Advantages - Flexibility - Scalability - Agility - Responsiveness - Real-time Interactions - Processing on-demand Disadvantages - More careful design required - More thought goes in design and tracking Now let us see the steps involved in Integrating Solace and MuleSoft. ## 1. Creating a Solace Cloud Trial Account You can create a Free Solace trial account using the following link: [https://console.solace.cloud/login](https://console.solace.cloud/login) ![Solace PubSub+ Cloud login page with email and password fields](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-1.png) Click on “Sign Up” and fill in all the details. You will receive a welcome email in your inbox. Now you will be able to sign in with your credentials in the login page. ![Solace 60-day free trial sign-up form with name and password fields](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-2.png) ## 2. Creating the Messaging Service Once you login with your credentials, you will see a welcome screen like the following one. ![PubSub+ Cloud welcome console showing Event Management and Event Streaming tiles](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-3.png) Now click on “Cluster Manager,” located at the left side of the screen to create a Service. According to the [official Solace documentation for Managing Services](https://docs.solace.com/Configuring-and-Managing/Managing-Services.htm#mc-main-content) - Solace PubSub+ event brokers support the following Services: - Solace Message Format (SMF)—This Service allows clients to communicate with an event broker on the Message Backbone interface. - Solace Element Management Protocol (SEMP)—This Service allows management applications to communicate with an event broker on the Management interface. - Web transport—This Service allows Web clients to communicate with an event broker. - Solace Representational State Transfer (REST)—This Service allows REST clients to communicate with an event broker using standard HTTP requests. - Message Queuing Telemetry Transport (MQTT)—This Service allows clients to communicate with an event broker using the MQTT messaging protocol. - Advanced Message Queuing Protocol (AMQP)—This Service allows AMQP clients to communicate with an event broker using the AMQP open standard application layer protocol. Let us spin up a Service. First select the Service Type you want to use. There are two types of Services: - Enterprise: A highly available Service on dedicated infrastructure with customizable connections - Developer: A Service on dedicated infrastructure with 100 connections. In this case, I have selected “Developer.” ![Create Service screen with Enterprise and Developer service type options](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-4.png) Then, select the Cloud. In my case, I chose “Amazon Web Services,” or “AWS” for short. As you can see from the following screenshot, there are different clouds that you can choose from. Select any cloud depending on your need. ![Cloud picker dropdown listing AWS, Google Cloud, Microsoft Azure and Private Cloud](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-5.png) ![Create Service form with Amazon Web Services chosen and a region map](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-6.png) Select the Region. In my case, I chose “US East (N. Virginia)” because it’s closer to me. ![AWS region selector with US East N. Virginia highlighted on the world map](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-7.png) Give the Service a name. A good naming convention is to create it using the Cloud and the Environment name, like “sol-aws-dev.” You can check with your team if there are any standards to name the Messaging Services. ![Service Details step with broker version and service name sol-aws-dev entered](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-8.png) After that, select the Message VPN name and start the Service by clicking “Start Service.” Your Service should now be up and running. ![Cluster Manager Services list showing the sol-aws-dev developer service Running](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-9.png) ## 3. Configuring the Solace KeyStore in MuleSoft Once we have a Messaging Service, we’ll need to configure the KeyStore in MuleSoft in order to connect to the Service. Click on the Messaging Service to open the Services’ settings. Once inside, click on the “Connect” tab, located at the top of the screen, and select the protocol you want to use to connect the Service. In our POC, we use the “Solace Messaging” option. ![Connect tab listing client libraries: Solace Messaging, AMQP, MQTT and REST](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-10.png) After you select the protocol, you will see all the Connection Details, which will be used in MuleSoft (JMS Config) in order to connect to Solace. ![Solace Messaging connection details: username, password, VPN, host and Download PEM](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-11.png) As you can see from the screenshot above, you can download the TrustStore certificate by clicking on the “Download PEM” link. This PEM file must be converted to a JKS format. In order to convert this file, we’ll use the KeyStore Explorer tool *(you can download it here -* [https://keystore-explorer.org/downloads.html](https://keystore-explorer.org/downloads.html)*).* Open KeyStore Explorer and create a new JKS KeyStore Type as shown in the following screenshot. ![KeyStore Explorer New KeyStore Type dialog with JKS selected](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-12.png) To import the PEM certificate, select the “Import Trusted Certificate” option (command+T or Ctrl+T). ![KeyStore Explorer context menu with Import Trusted Certificate highlighted](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-13.png) Select the downloaded PEM certificate from Solace and give it a name to import it. ![KeyStore Explorer showing the imported digicert global root ca RSA certificate](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-14.png) Click on “File” and “Save As.” Enter the password of your choice and confirm. Please keep a note of the password you entered as you have to use this in the MuleSoft Configuration (JMS Config). You will see a screen similar to the next one to save the converted JKS file to your system. ![Save KeyStore As dialog saving DigicertCA.jks into a keystore folder](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-15.png) Now all you have to do is import this JKS file into your Mule Project. I put it under `src/main/resources/keystores/` but you can use it as long as it’s under `src/main/resources`. ![Studio Package Explorer with DigicertCA.jks under src/main/resources/keystores](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-16.png) ## 4. Configuring the Message Service In order to establish a connectivity between the clients and the event broker, we need to configure the Messaging Service as per our needs. Click on the “Manage” tab at the top of the screen, or next to the “Connect” tab that we just used. ![Manage tab showing Event Broker settings and PubSub+ Broker Manager quick settings](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-17.png) Locate and click the “Message VPN” button in the middle, under the PubSub+ Broker Manager Quick Settings label. A new window will open. ![PubSub+ Broker Manager Message VPN summary dashboard with usage gauges](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-18.png) If you click on the “Services” tab, you will see all the Services that are enabled. ![Message VPN Services tab listing REST settings, ports and client connection limits](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-19.png) Here are the settings that I created for the Proof of Concept. You can configure your own settings or based on mine if you’re experimenting. You can also modify the message VPN settings by clicking on the “Show Advanced Settings” as shown in the following screenshot. ![Message VPN Settings tab with the Show Advanced Settings link circled](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-20.png) In the “Stats” you will be able to see the number of messages published in your message VPN, the number of messages received, etc. ![Message VPN Stats tab showing message counts and data rate statistics](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-21.png) ### Client Connections There is no need to explicitly set the client connections since all these settings are pre-filled. ![Client Connections list showing connected Solace clients and their addresses](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-22.png) ### Access Control - Client Authentication This tab contains details about what authentication method you wish to use. By default “Basic Authentication” is enabled. If you want to use the “Client Certificate Authentication,” you could enable that. In my case I am using “Basic Authentication.” You could also see the settings of “Client Profiles,” “ACL Profiles” and “Client Usernames” as shown in the following screenshots. ![Access Control Client Authentication tab with Basic Authentication enabled](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-23.png) ### Access Control - Client Usernames ![Client Usernames tab listing default and solace-cloud-client usernames](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-24.png) ## 5. Configuring Queues Guaranteed messages in Solace PubSub+ are stored in event broker endpoints. There are two types of endpoints: queues and topic endpoints. Topic endpoints are used by JMS (that is, durable topic subscriptions). *To understand the basics of Messaging, exchange Patterns, Queues and Topics, please refer to this site:* [https://docs.solace.com/PubSub-Basics/Message-What-Is.htm](https://docs.solace.com/PubSub-Basics/Message-What-Is.htm)*.* I have created the following 3 Queues by clicking on the “+ Queue” button as shown in the screenshot. ![Queues list with sample-test-queue, error-queue and dead-letter-queue, all durable](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-25.png) ### Differences between Queues and Topics Queues: - Allow one-to-one communication (Point-to-Point). - Typically used when you want to send a single event to exactly one consumer. Topics: - Allow one-to-many communication (Publish-Subscribe). - Typically used when you want to send a single event to multiple consumers. I have created a Topic Subscription for each Queue as shown in the following screenshots. *To understand why we need to map a Topic to a Queue please refer to* [https://docs.solace.com/PubSub-Basics/Core-Concepts-Endpoints-Queues.htm](https://docs.solace.com/PubSub-Basics/Core-Concepts-Endpoints-Queues.htm)*.* ![Subscriptions tab for sample-test-queue mapped to sample-test-topic](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-26.png) ![Subscriptions tab for sample-test-error-queue mapped to sample-test-error-topic](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-27.png) ![Subscriptions tab for the dead-letter-queue mapped to sample-test-dead-letter-topic](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-28.png) ## 6. Configuring the MuleSoft Project In the MuleSoft Project, the following dependencies must be added in the `pom.xml` file. There are three ways to do this. You can add the modules directly into the pom, you can add them through Exchange from the Mule Palette, or you can right click on the project > Manage Dependencies > Manage (or Add) Modules. We need to add the Solace and JMS modules to develop our Mule Application. After adding these modules, the JMS connectors will be available in the Mule Palette: ![Studio Mule Palette searching JMS, showing Ack, Consume, Publish operations](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-29.png) Once you have these, create a Global Element for the JMS Configuration with your desired settings and the connection information from Solace (as mentioned in the “Configuring the Solace KeyStore in MuleSoft” section). For more information on the JNDI Connection properties refer to [https://docs.solace.com/Solace-JMS-API/JNDI-Connection-Properti.htm#mc-main-content](https://docs.solace.com/Solace-JMS-API/JNDI-Connection-Properti.htm#mc-main-content) Here’s an example: > [!NOTE] > The GitHub repository with all this code can be found at the end of the post. You can double-check the settings and properties there if you have some doubts. ![JMS Config global element with connection factory and JNDI name resolver settings](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-30.png) ![JMS Config provider properties mapping Solace JNDI and SSL truststore values](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-31.png) ## 7. Publishing Message to Solace Topic In my case, I added an HTTP Listener in my Mule flow to be able to listen to a request with the content that I want to publish into the Solace Topic. Here you can see how I have configured the JMS Publish connector: ![Mule publish flow with Listener, JMS Publish to Solace Topic and a Logger](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-32.png) ## 8. Testing the “Happy Path” Start the Mule App in your local to make sure that everything was correctly configured. Once the app is up and running, send a POST request from [Postman](https://www.postman.com/downloads/) (you can also use the [Advanced REST Client](https://install.advancedrestclient.com/install), [cURL](https://curl.se/), or any other tool to send a request to your local app). The Mule app will read the payload you send here and will publish it into the Solace Topic. ![Postman POST to localhost:8081/publish with a JSON account body](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-33.png) For this demo, I sent a JSON Body like this one: ```json [ { "name":"Elon Musk", "billingAddress":"Royalwest Drive", "billingCity":"Toronto", "billingCountry":"USA", "billingPostalCode":"mp3lp7", "billingState":"NY" } ] ``` As you can see in the following screenshot, the message published to the Topic can be seen in the Solace Queue mapped to it. ![Messages Queued tab for sample-test-queue showing one spooled 353-byte message](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-34.png) ## 9. Subscribing a Message from the Solace Queue The queued messages must now be processed. In my case, the message that is queued contains information from a Salesforce Account. I have to subscribe to this queue in a flow, transform it, and create an Account object in SFDC. To achieve this, I have created a new flow called “solace-jms-subscribe.” ![Mule subscribe flow consuming the queue, transforming and creating an SFDC Account](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-35.png) In the “On New Message” component settings, please note that the “Acknowledgement Mode” is set to “Manual” since we want to delete the message in the queue only after further processing it and sending the positive Acknowledgement. ![On New Message JMS settings with Queue consumer and MANUAL acknowledge mode](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-36.png) There are different Acknowledgement modes as shown. *More information on all the settings can be found in* [https://docs.mulesoft.com/jms-connector/1.7/jms-ack](https://docs.mulesoft.com/jms-connector/1.7/jms-ack)*.* ![Acknowledge Mode dropdown listing AUTO, DUPS_OK, IMMEDIATE and MANUAL options](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-37.png) When the project is deployed in Anypoint Studio, this message is automatically consumed by the “On New Message” JMS Subscriber as shown in the following screenshot: ![Mule Debugger showing the consumed message payload and the deployed app console](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-38.png) The message is then transformed to the required format. ![Transform Message DataWeave mapping queue fields to Salesforce account fields](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-39.png) And the account is created in Salesforce successfully using the “Create” Connector. After this, a positive Acknowledgement is sent back to Solace using the “Positive Ack” JMS connector. ![Salesforce All Accounts list including the newly created Elon Musk account](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-40.png) The message in the Queue is now deleted: ![Messages Queued tab for sample-test-queue now showing zero messages queued](../../assets/blog/how-to-integrate-solace-pubsub-cloud-with-mulesoft-41.png) Thank you so much for reading my post and I hope it will be of some use. In the next post we'll dive into the negative scenario and error handling. Happy Learning :) Pravallika Nagaraja ## GitHub repository [ProstDev GitHub - Solace MuleSoft Integration](https://github.com/ProstDev/solace-mulesoft-integration) --- ## The Power of cURL Source: https://prostdev.com/post/the-power-of-curl | Published: Mar 30, 2021 | Category: Tutorials [Advanced Rest Client](https://install.advancedrestclient.com/install), [Postman](https://www.postman.com/downloads/) and [SoapUI](https://www.soapui.org/downloads/soapui/) are top API clients you may have used while building or testing API integrations. These applications are useful. However there are cases where invoking the API from the command line is advantageous and is made possible through **cURL.** ## What is cURL? [cURL](https://curl.se/) is a tool used to transfer files. The cURL command can be used inside scripts or from the command line. cURL provides support for common protocols like HTTP, HTTPS, FTP and much more. This article will focus on using the cURL command to invoke integrations that use HTTP. The cURL commands provided in this article will be demonstrated by invoking a sample API built in Mule 4, running on my local machine. Let’s get started! ## How do I start? Open your command prompt (Windows) or terminal (Mac/Linux). Run ```bash curl ``` ![Git Bash terminal where running curl prints "try 'curl --help' for more information"](../../assets/blog/the-power-of-curl-1.png) Look for the following output: ```bash curl: try 'curl --help' for more information ``` This is a good indicator that cURL is installed on your local machine. If not, you will need to do a fresh install of cURL. Download cURL [here](https://curl.se/download.html). cURL commands typically use the following syntax: ```bash curl [options] [url] ``` ## Need help? Use the --help option to provide a list of options. ```bash curl --help ``` ![curl --help output listing common options like --data, --header, --output, and --verbose](../../assets/blog/the-power-of-curl-2.png) Use the --manual option to reference the “man” page (or manual). ```bash curl --manual ``` ![curl --manual page showing the cURL name, synopsis, and description of supported protocols](../../assets/blog/the-power-of-curl-3.png) > [!NOTE] > Since I am using a Windows machine, I often use the command line from a Git Bash terminal. ## Demo Time The remainder of this article will focus on HTTP. All demonstrated cURL command examples will be HTTP requests (unless otherwise specified) and invoked against a simple Mule API running on my local machine. The API takes in the HTTP request and provides a breakdown of the request in the response. REQUEST OPTION: To create an HTTP/HTTPS request. Use the --request option. SYNTAX: ```bash curl --request '' ``` > [!NOTE] > URL will include the protocol (http/https). ### **GET** GET requests can be written in two formats: 1) Short-hand SYNTAX: ```bash curl '' ``` EX: ```bash curl 'http://localhost:8081/curl/helloget' ``` 2) Long-hand SYNTAX: ```bash curl --request GET '' ``` EX: ```bash curl --request GET 'http://localhost:8081/curl/helloget' ``` ![curl GET request to /curl/helloget and the JSON response echoing the request method and path](../../assets/blog/the-power-of-curl-4.png) ### **QUERY PARAMS** SYNTAX: ```bash curl --request '?=...&=' ``` EX: ```bash curl --request GET 'http://localhost:8081/curl/helloget?firstname=Daffy&lastname=Duck' ``` ![curl GET with firstname and lastname query params echoed back in the response queryParams](../../assets/blog/the-power-of-curl-5.png) ### **URI PARAMS** SYNTAX: ```bash curl --request '/' ``` EX: ```bash curl --request GET 'http://localhost:8081/curl/helloget/1' ``` ![curl GET to /curl/helloget/1 with the uriParams id of 1 echoed in the response](../../assets/blog/the-power-of-curl-6.png) ### **HEADERS** SYNTAX: ```bash curl --request '' \ --header ':' ``` *Single Header* EX: ```bash curl --request GET 'http://localhost:8081/curl/helloget' \ --header 'Authorization: Bearer xybaneoovganfZbe' ``` ![curl GET with a single Authorization Bearer header echoed in the response headers](../../assets/blog/the-power-of-curl-7.png) *Multiple Headers* EX: ```bash curl --request GET 'http://localhost:8081/curl/helloget' \ --header 'client_id: bond' \ --header 'client_secret: 0007' ``` ![curl GET with client_id and client_secret headers echoed in the response headers](../../assets/blog/the-power-of-curl-8.png) ### **POST** SYNTAX: ```bash curl --request '' \ --header 'Content-Type: ' \ --data-raw '' ``` EX: ```bash curl --request POST 'http://localhost:8081/curl/hellopost' \ --header 'Content-Type: application/json' \ --data-raw '{ "RQ" : {"message" : "hello prostdev readers"}}' ``` ![curl POST to /curl/hellopost with a JSON body echoed back in the response requestBody](../../assets/blog/the-power-of-curl-9.png) ### **Print Output** Interested in saving the output of your API invocation? Use the --output option. SYNTAX: ```bash curl --request --output output.txt ``` EX: ```bash curl --request GET 'http://localhost:8081/curl/helloget'\ --output output.txt ``` ![curl GET request using --output output.txt to save the response to a file](../../assets/blog/the-power-of-curl-10.png) ![The saved output.txt opened in Notepad showing the JSON response from the curl request](../../assets/blog/the-power-of-curl-11.png) ## Lessons Learned - Although quotes are not required around the URL, when sending requests with query parameters make sure you wrap requests around quotes. Without the quotes, I noticed that cURL dropped the query parameters. - Wrap your request URL in “double-quotes” if using command prompt and ‘single-quotes’ if using command line terminal. I found issues with ‘’ and “” between command prompt and command line. - Multi-line in command prompt is different from multi-line on command line. - Add “\” at the end of the line to enter a new line on a multi-line request from the command line. - Add “^” at the end of the line to enter a new line on a multi-line request from the command prompt. ## Conclusion This post invoked an API through HTTP. However, most of the time we invoke APIs over HTTPS. If you are interested in an HTTPS version of this post, please let us know through a comment. If you would like to learn more about cURL, check out the cURL documentation [here](https://curl.se/docs/httpscripting.html). Lastly, I am Whitney Akinola. I’m a fellow MuleSoft Developer and content creator. Feel free to check out my technical content [here](https://www.whitneyakinola.io/blog). --- ## How to check for empty values in an array in DataWeave | Part 4: Arrays Module Source: https://prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-4-arrays-module | Published: Feb 23, 2021 | Category: Tutorials I had this use-case where I had to make sure all the values inside an [array](https://en.wikipedia.org/wiki/Array_data_structure) were not empty. By “empty,” I’m referring to an empty string (“”), a null value (*null*), or even an empty array ([]). There were some cases when I would receive a null value instead of an array. The array could only contain strings or nulls, but not objects. However, I’ll be adding tests to check for empty objects as well. > [!NOTE] > The term “null” refers to a null value, whereas “Null” refers to the data type. The same rule applies to string/String, array/Array, and object/Object. I can have an empty array ([]), which is of the type Array, or a string “abc,” which is of the type String. In this series of posts, I explain 6 different approaches to achieve (almost) the same output using different DataWeave functions/operators. As we advance through the posts, the functions will become easier to read, and the logic will have fewer flaws. - [Part 1: Using sizeOf, groupBy, isEmpty, and default](https://www.prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-1-sizeof-groupby-isempty-default) - This solution had some readability issues, the code was showing a warning, and it didn’t fully work for all of our use cases. - [Part 2: Using sizeOf, filter, isEmpty, and default](https://www.prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-2-sizeof-filter-isempty-default) - This solution still had some readability issues, and it still didn’t fully work for all of our use cases. - [Part 3: Using isEmpty and filter](https://www.prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-3-isempty-filter) - This solution worked as expected, but we can do better. ;) - **Part 4: Using the Arrays Module, Pattern Matching, and Function Overloading.** I use the DataWeave Playground throughout these articles. You can follow [this post](https://www.prostdev.com/post/how-to-run-locally-the-dataweave-playground-docker-image) to set up a local Docker image (no previous Docker experience is needed), or you can also open a new Mule project and use the Transform Message component. > [!NOTE] > You can also use the online DataWeave Playground tool from [this link](http://dwlang.fun/), if available. ## Using the Arrays Module I’ll skip the step-by-step process that we’ve been doing for the past 3 posts to show you 3 different approaches this time. Let’s take a look at the function we created in the last post. ```dataweave %dw 2.0 output application/json fun containsEmptyValues(arr) = if (isEmpty(arr)) true else not isEmpty(arr filter isEmpty($)) --- { nullValue: containsEmptyValues(null), emptyArray: containsEmptyValues([]), arrayWithEmptyString: containsEmptyValues([""]), arrayWithNull: containsEmptyValues([null]), arrayWithEmptyString2: containsEmptyValues(["1", ""]), arrayWithNull2: containsEmptyValues(["1", null]), arrayWithValues: containsEmptyValues(["1", "2"]), arrayWithEmptyObject: containsEmptyValues([{}]), arrayWithEmptyObject2: containsEmptyValues(["1",{}]), arrayWithNonEmptyObject: containsEmptyValues([{a:"b"}]) } ``` ![Part 3 isEmpty/filter function tested against ten cases, all returning the expected results](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-4-arrays-module-1.png) We will be using the “[some](https://docs.mulesoft.com/mule-runtime/4.3/dw-arrays-functions-some)” function from the [Arrays module](https://docs.mulesoft.com/mule-runtime/4.3/dw-arrays). To do this, we first need to import the function from the array. ```dataweave %dw 2.0 output application/json import some from dw::core::Arrays fun containsEmptyValues(arr) = if (isEmpty(arr)) true else not isEmpty(arr filter isEmpty($)) --- ``` > [!NOTE] > You can also import all the functions from a module using “import from…” After that, we’ll replace the “else” expression with “arr some isEmpty($)”. This will check if at least one of the items inside the “arr” array is an empty value. If this is the case, it will return a true value. ```dataweave %dw 2.0 output application/json import some from dw::core::Arrays fun containsEmptyValues(arr) = if (isEmpty(arr)) true else arr some isEmpty($) --- ``` ![Function using the Arrays module some function, producing the same correct ten-case results](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-4-arrays-module-2.png) As you can see, we receive the same expected output. We didn’t remove the first condition from the code because when we try to do “arr some isEmpty($)” when arr is null or when arr is an empty array ([]), the output will be false instead of the true value that we’re expecting. ![some isEmpty on null and on an empty array both returning false instead of true](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-4-arrays-module-3.png) ## Using Pattern Matching (match/case) This alternative uses [pattern matching](https://docs.mulesoft.com/mule-runtime/4.3/dataweave-pattern-matching) with the match/case statements (sort of like the switch/case Java statements). This is what our code would look like: ```dataweave %dw 2.0 output application/json import some from dw::core::Arrays fun containsEmptyValues(arr) = arr match { case [] -> true case a is Array -> a some isEmpty($) else -> isEmpty(arr) } --- ``` ![match/case version of the function with correct results across the ten test cases](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-4-arrays-module-4.png) In this case, we’re immediately returning true when we receive an empty array ([]). If our data is not an empty array, then we go to the second “case,” which checks if the parameter is of type Array and runs the logic that includes the “some” function. Finally, if none of those two conditions match (when the value is null), we go to the “else” part and run the “isEmpty” function. ## Using Function Overloading [Function overloading](https://docs.mulesoft.com/mule-runtime/4.3/dataweave-functions#overloading-functions) executes pattern matching logic underneath. If performance is an issue, use this solution instead of the previous one. This is what our code would look like for our two types (Null and Array): ```dataweave %dw 2.0 output application/json import some from dw::core::Arrays fun containsEmptyValues(value: Null) = true // first fun fun containsEmptyValues(arr: Array) = ( // second fun if (arr == []) true // if (isEmpty(arr)) true else arr some isEmpty($) ) --- ``` ![Function-overloading version with separate Null and Array functions, correct ten-case results](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-4-arrays-module-5.png) We’re defining the same function twice, both with just one parameter. The difference is that the first one expects a Null data type, and the second one expects Array. By making this distinction, we can immediately return a true value if the parameter is a null value (the first function) and the rest of the logic is handled by the second function. Note that you can define the condition either as “if (arr == []) true” or as “if (isEmpty(arr)) true”, and both will work as expected. You can also use some pattern matching under the second function instead of using an if/else statement. This code would look something like this: ```dataweave fun containsEmptyValues(arr: Array) = arr match { //second fun case [] -> true else -> arr some isEmpty($) } ``` ## Recap Let’s finalize this series with a summary of all the approaches we have learned throughout these posts. **Solution 1: sizeOf, groupBy, isEmpty, default** ```dataweave fun containsEmptyValues(arr) = sizeOf((arr groupBy isEmpty($))."true" default []) > 0 ``` - Not very easy to read (too many parentheses). - DataWeave shows a warning. - Null and empty array return false instead of true. **Solution 2: sizeOf, filter, isEmpty, default** ```dataweave fun containsEmptyValues(arr) = sizeOf(arr default [] filter isEmpty($)) > 0 ``` - Slightly easier to read, but still confusing. - Null and empty array return false instead of true. **Solution 3: isEmpty, filter** ```dataweave fun containsEmptyValues(arr) = if (isEmpty(arr)) true else not isEmpty(arr filter isEmpty($)) ``` - Works as expected, but may not be the best solution for larger payloads. **Solution 4: Arrays Module** ```dataweave import some from dw::core::Arrays fun containsEmptyValues(arr) = if (isEmpty(arr)) true else arr some isEmpty($) ``` - Works as expected, and the performance might be better for larger payloads. **Solution 5: Pattern Matching** ```dataweave import some from dw::core::Arrays fun containsEmptyValues(arr) = arr match { case [] -> true case a is Array -> a some isEmpty($) else -> isEmpty(arr) } ``` - Works as expected and leaves room to implement more “case” statements to handle additional data types or conditions. **Solution 6: Function Overloading** ```dataweave import some from dw::core::Arrays fun containsEmptyValues(value: Null) = true // 1) using if/else fun containsEmptyValues(arr: Array) = ( if (arr == []) true // if (isEmpty(arr)) true else arr some isEmpty($) ) // 2) using match/case // fun containsEmptyValues(arr: Array) = arr match { // case [] -> true // else -> arr some isEmpty($) // } ``` - Works as expected and leaves room to implement more functions to handle additional data types. - Function overloading executes pattern matching logic underneath. If performance is an issue, use this solution instead of solution 5. Oh! And before I forget, there was a **quiz** in the previous post. The question was, “Do the following scripts generate the same output?” ```dataweave 1) not true or true 2) ! true or true ``` The answer is: NO! They do *not* return the same value. The first one returns false, while the second one returns true. Why? This is how these scripts are really run: ```dataweave 1) not (true or true) = not (true) = false 2) (! true) or true = (false) or true = true ``` Mind-blowing, right?! You can read more about this in the [Mule Docs - Logical Operators](https://docs.mulesoft.com/mule-runtime/4.3/dw-operators#logical_operators). And now we’re really done with this series! Which solution would you choose? Don’t miss any DataWeave content! Subscribe to our newsletter or follow us on social media to become a DataWeave champion. :D Keep on practicing! -Alex --- ## How to check for empty values in an array in DataWeave | Part 3: isEmpty, filter Source: https://prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-3-isempty-filter | Published: Feb 16, 2021 | Category: Tutorials I had this use-case where I had to make sure all the values inside an [array](https://en.wikipedia.org/wiki/Array_data_structure) were not empty. By “empty,” I’m referring to an empty string (“”), a null value (*null*), or even an empty array ([]). There were some cases when I would receive a null value instead of an array. The array could only contain strings or nulls, but not objects. However, I’ll be adding tests to check for empty objects as well. > [!NOTE] > The term “null” refers to a null value, whereas “Null” refers to the data type. The same rule applies to string/String, array/Array, and object/Object. I can have an empty array ([]), which is of the type Array, or a string “abc,” which is of the type String. In this series of posts, I explain 6 different approaches to achieve (almost) the same output using different DataWeave functions/operators. As we advance through the posts, the functions will become easier to read, and the logic will have fewer flaws. - [Part 1: Using sizeOf, groupBy, isEmpty, and default.](https://www.prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-1-sizeof-groupby-isempty-default) - [Part 2: Using sizeOf, filter, isEmpty, and default.](https://www.prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-2-sizeof-filter-isempty-default) - **Part 3: Using isEmpty and filter.** - [Part 4: Using the Arrays Module, Pattern Matching, and Function Overloading.](https://www.prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-4-arrays-module) I use the DataWeave Playground throughout these articles. You can follow [this post](https://www.prostdev.com/post/how-to-run-locally-the-dataweave-playground-docker-image) to set up a local Docker image (no previous Docker experience is needed), or you can also open a new Mule project and use the Transform Message component. > [!NOTE] > You can also use the online DataWeave Playground tool from [this link](http://dwlang.fun/), if available. ## Building the solution I’ll create an array to start building and testing this solution: ```dataweave %dw 2.0 output application/json --- ["notEmpty", "", null] ``` **Step 1**: Just as we did in the previous solution, we’ll use the “filter” function, followed by “isEmpty” to retrieve the empty items from the array. ```dataweave ["notEmpty", "", null] filter isEmpty($) ``` ![DataWeave Playground filtering ["notEmpty","",null] by isEmpty, outputting just the "" and null entries.](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-3-isempty-filter-1.png) **Step 2**: This time, we will not be counting the values with “sizeOf.” Instead, let’s use another “isEmpty” function for this filtered array ([“”, null]). This function can let us know if the array contains any value or an empty array ([]). ```dataweave isEmpty(["notEmpty", "", null] filter isEmpty($)) ``` ![DataWeave Playground wrapping the filtered array in isEmpty, which outputs false.](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-3-isempty-filter-2.png) With this, we know that the original array ([“notEmpty”, “”, null]) indeed contains empty values (because it’s not empty after filtering it). **Step 3**: Although our goal was to receive a true value if there were empty values, not a false, right? Not to worry! We can simply negate the whole thing with the “[not](https://docs.mulesoft.com/mule-runtime/4.3/dw-operators#logical_operators)” operator. ```dataweave not isEmpty(["notEmpty", "", null] filter isEmpty($)) ``` ![DataWeave Playground negating the result with not isEmpty, which outputs true.](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-3-isempty-filter-3.png) **Quiz**: Do the following scripts generate the same output? ```dataweave 1) not true or true 2) ! true or true ``` *The answer will be revealed in the next post...Surprise! - There will be more! :)* And now…*drumroll*...we get to the test that broke our two previous solutions…What if we send a null value instead of an array? ```dataweave not isEmpty(null filter isEmpty($)) ``` ![DataWeave Playground running not isEmpty(null filter isEmpty($)) without error, outputting false.](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-3-isempty-filter-4.png) It didn’t break this time! Woohoo! Since we’re no longer using the “[sizeOf](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-sizeof)” function, which only accepts Array, Object, Binary, and String data types (not Null), we don’t have that issue anymore! “[isEmpty](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-isempty)” is cool with Array, String, Null, and Object. Look at that! We’re done! We’re already returning a Boolean, so there’s no need to create a condition. Now we just have to make this a function and call it with our test data. ## Setting up test values To create the function, we can simply define it over the 3 dashes (---) using the “fun” keyword. We can copy and paste our previous code for this new function and replace the [hardcoded](https://en.wikipedia.org/wiki/Hard_coding) array (or null value) with the function’s argument. Like this: ```dataweave %dw 2.0 output application/json fun containsEmptyValues(arr) = not isEmpty(arr filter isEmpty($)) --- ``` To test our use-cases, we’ll be using the same payload we created in our previous solutions: ```dataweave %dw 2.0 output application/json fun containsEmptyValues(arr) = not isEmpty(arr filter isEmpty($)) --- { nullValue: containsEmptyValues(null), emptyArray: containsEmptyValues([]), arrayWithEmptyString: containsEmptyValues([""]), arrayWithNull: containsEmptyValues([null]), arrayWithEmptyString2: containsEmptyValues(["1", ""]), arrayWithNull2: containsEmptyValues(["1", null]), arrayWithValues: containsEmptyValues(["1", "2"]), arrayWithEmptyObject: containsEmptyValues([{}]), arrayWithEmptyObject2: containsEmptyValues(["1",{}]), arrayWithNonEmptyObject: containsEmptyValues([{a:"b"}]) } ``` ![containsEmptyValues function tested against ten cases; nullValue and emptyArray wrongly return false.](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-3-isempty-filter-5.png) But the “nullValue” and “emptyArray” fields are returning false instead of the true value that we need. **Step 4**: Let’s add a condition to check if the argument (arr) is empty and to immediately return true if that’s the case. Don’t forget to add the “else” with the logic we already had. ```dataweave fun containsEmptyValues(arr) = if (isEmpty(arr)) true else not isEmpty(arr filter isEmpty($)) ``` ![Fixed containsEmptyValues using an if (isEmpty(arr)) guard, now returning true for nullValue and emptyArray.](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-3-isempty-filter-6.png) Oh, look at that! We did it!! ## Does it work as expected? It works for ALL of our use-cases! But there are more ways to achieve the same output. Have you heard of function overloading or pattern matching using match/case? We can also achieve this solution with these two additional approaches. Plus, we will review how to use the Arrays module! Remember that the answer to the **quiz** will be revealed in the next post. Here is the question: Do the following scripts generate the same output? - not true or true - ! true or true Subscribe now to receive a notification as soon as the next content is published! *Prost!* -Alex --- ## How to check for empty values in an array in DataWeave | Part 2: sizeOf, filter, isEmpty, default Source: https://prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-2-sizeof-filter-isempty-default | Published: Feb 9, 2021 | Category: Tutorials I had this use-case where I had to make sure all the values inside an [array](https://en.wikipedia.org/wiki/Array_data_structure) were not empty. By “empty,” I’m referring to an empty string (“”), a null value (*null*), or even an empty array ([]). There were some cases when I would receive a null value instead of an array. The array could only contain strings or nulls, but not objects. However, I’ll be adding tests to check for empty objects as well. > [!NOTE] > The term “null” refers to a null value, whereas “Null” refers to the data type. The same rule applies to string/String, array/Array, and object/Object. I can have an empty array ([]), which is of the type Array, or a string “abc,” which is of the type String. In this series of posts, I explain 6 different approaches to achieve (almost) the same output using different DataWeave functions/operators. As we advance through the posts, the functions will become easier to read, and the logic will have fewer flaws. - [Part 1: Using sizeOf, groupBy, isEmpty, and default.](https://www.prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-1-sizeof-groupby-isempty-default) - **Part 2: Using sizeOf, filter, isEmpty, and default.** - [Part 3: Using isEmpty and filter.](https://www.prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-3-isempty-filter) - [Part 4: Using the Arrays Module, Pattern Matching, and Function Overloading.](https://www.prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-4-arrays-module) I use the DataWeave Playground throughout these articles. You can follow [this post](https://www.prostdev.com/post/how-to-run-locally-the-dataweave-playground-docker-image) to set up a local Docker image (no previous Docker experience is needed), or you can also open a new Mule project and use the Transform Message component. > [!NOTE] > You can also use the online DataWeave Playground tool from [this link](http://dwlang.fun/), if available. ## Building the solution I’ll create an array to start building and testing this solution: ```dataweave %dw 2.0 output application/json --- ["notEmpty", "", null] ``` **Step 1**: We’ll use the “filter” function, followed by “isEmpty” to retrieve the empty items from the array. ```dataweave ["notEmpty", "", null] filter isEmpty($) ``` ![DataWeave filter isEmpty script returning an array of the empty string and null](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-2-sizeof-filter-isempty-default-1.png) Compared to the solution in the previous post, which was using “groupBy” to retrieve all the empty values, we can get the same result in one step instead of two. **Step 2**: As we did in the previous solution, we need to count how many values are in this array ([“”, null]). We’ll be using the “sizeOf” function as well. ```dataweave sizeOf(["notEmpty", "", null] filter isEmpty($)) ``` ![sizeOf wrapping the filter isEmpty script, outputting 2](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-2-sizeof-filter-isempty-default-2.png) Will you look at that? There’s no green line! In the solution prior, we were getting a warning from DataWeave saying that it was auto-coercing the data type. Well, not this time, DataWeave! Although…We do need to test what would happen if we sent Null instead of Array. Let’s see. ```dataweave sizeOf(null filter isEmpty($)) ``` ![DataWeave error: unable to call sizeOf with a Null argument from filtering null](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-2-sizeof-filter-isempty-default-3.png) And…That’s an error. :( When you try to filter a null value in this case, the result is also null. So when we try to call the “[sizeOf](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-sizeof)” function with Null, it fails because it only accepts Array, Object, Binary, and String data types as arguments. How can we fix this? We use the “default” operator. Pretty much like in the previous solution, but this time in a different place. ```dataweave sizeOf(null default [] filter isEmpty($)) ``` This operator will check for any null value and default the value to an empty array ([]). Now the “filter” will receive the empty array and will also output an empty array. The “sizeOf” function does work on empty arrays, and we end up with a zero (which is what we expect). ![sizeOf with null default [] filter isEmpty now returning 0 without error](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-2-sizeof-filter-isempty-default-4.png) Now let’s put back the array we had before, instead of the null value, and continue to the next step. ```dataweave sizeOf(["notEmpty", "", null] default [] filter isEmpty($)) ``` **Step 3**: We want to return a Boolean depending on whether there are empty values in the array or not. Let’s add a condition to return true when there *are* empty values or false if there aren’t any. ```dataweave if ( sizeOf(["notEmpty", "", null] default [] filter isEmpty($)) > 0 ) true else false ``` ![if-condition script checking sizeOf greater than 0, returning true](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-2-sizeof-filter-isempty-default-5.png) We got what we wanted! Now we just have to make this a function and call it with different values to make sure that it works for all our cases. ## Setting up test values To create the function, we can simply define it over the 3 dashes (---) using the “fun” keyword. We can copy and paste our previous code for this new function, and replace the [hardcoded](https://en.wikipedia.org/wiki/Hard_coding) array with the function’s argument. Like this: ```dataweave %dw 2.0 output application/json fun containsEmptyValues(arr) = if ( sizeOf(arr default [] filter isEmpty($)) > 0 ) true else false --- ``` To test our use-cases, we’ll be using the same payload we created in our previous solution: ```dataweave %dw 2.0 output application/json fun containsEmptyValues(arr) = if ( sizeOf(arr default [] filter isEmpty($)) > 0 ) true else false --- { nullValue: containsEmptyValues(null), emptyArray: containsEmptyValues([]), arrayWithEmptyString: containsEmptyValues([""]), arrayWithNull: containsEmptyValues([null]), arrayWithEmptyString2: containsEmptyValues(["1", ""]), arrayWithNull2: containsEmptyValues(["1", null]), arrayWithValues: containsEmptyValues(["1", "2"]), arrayWithEmptyObject: containsEmptyValues([{}]), arrayWithEmptyObject2: containsEmptyValues(["1",{}]), arrayWithNonEmptyObject: containsEmptyValues([{a:"b"}]) } ``` ![containsEmptyValues function tested against ten cases, with true/false results per case](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-2-sizeof-filter-isempty-default-6.png) Note that there are 3 fields at the end to test the behavior of the function with Object. Even though this was not our initial use-case, our “containsEmptyValues” function works with the Object data type as well. Why? Because we’re using the “[isEmpty](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-isempty)” function, which accepts the Array, String, Null, and Object data types. There was a **quiz** in the previous post. The question was, “*What happens when we change the “isEmpty” function to “*[isBlank](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-isblank)*” instead?*” and that same question can also be asked for this solution. The **answer** is that there is an error whenever you try to pass a value that’s not String or Null. In other words, the last 3 fields (arrayWithEmptyObject, arrayWithEmptyObject2, and arrayWithNonEmptyObject) would fail because we’d be sending an object ({}) instead of a string or a null value inside the array. ## Does it work as expected? It works for *most* of our use-cases. However, it’s not the best solution. Here are some of the cons of this function: - It’s slightly easier to read than the previous solution. However, there are still some parentheses that may cause confusion. - We still have the same problem as last time. The null value (null) and the empty array ([]) are returning a false value, but they should be returning a true. And that’s it for the second function! Remember that the following post will explain a more elegant solution, so subscribe now to receive an email as soon as the next part is published! Next time there will be dramatic changes, you’ll see! *Prost!* -Alex --- ## How to check for empty values in an array in DataWeave | Part 1: sizeOf, groupBy, isEmpty, default Source: https://prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-1-sizeof-groupby-isempty-default | Published: Feb 2, 2021 | Category: Tutorials I had this use-case where I had to make sure all the values inside an [array](https://en.wikipedia.org/wiki/Array_data_structure) were not empty. By “empty,” I’m referring to an empty string (“”), a null value (*null*), or even an empty array ([]). There were some cases when I would receive a null value instead of an array. The array could only contain strings or nulls, but not objects. However, I’ll be adding tests to check for empty objects as well. > [!NOTE] > The term “null” refers to a null value, whereas “Null” refers to the data type. The same rule applies to string/String, array/Array, and object/Object. I can have an empty array ([]), which is of the type Array, or a string “abc,” which is of the type String. In this series of posts, I explain 6 different approaches to achieve (almost) the same output using different DataWeave functions/operators. As we advance through the posts, the functions will become easier to read, and the logic will have fewer flaws. - **Part 1: Using sizeOf, groupBy, isEmpty, and default.** - [Part 2: Using sizeOf, filter, isEmpty, and default.](https://www.prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-2-sizeof-filter-isempty-default) - [Part 3: Using isEmpty and filter.](https://www.prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-3-isempty-filter) - [Part 4: Using the Arrays Module, Pattern Matching, and Function Overloading.](https://www.prostdev.com/post/how-to-check-for-empty-values-in-an-array-in-dataweave-part-4-arrays-module) I use the DataWeave Playground throughout these articles. You can follow [this post](https://www.prostdev.com/post/how-to-run-locally-the-dataweave-playground-docker-image) to set up a local Docker image (no previous Docker experience is needed), or you can also open a new Mule project and use the Transform Message component. > [!NOTE] > You can also use the online DataWeave Playground tool from [this link](http://dwlang.fun/), if available. ## Building the solution I’ll create an array to start building and testing this first solution. Something like this: ```dataweave %dw 2.0 output application/json --- ["notEmpty", "", null] ``` **Step 1**: We’ll use the “groupBy” function to separate the non-empty and empty items from the array (using the “isEmpty” function). ```dataweave ["notEmpty", "", null] groupBy isEmpty($) ``` This will give us an object with two fields: one “false” and one “true.” The “false” field contains an array with the values that are not empty ([“notEmpty”]), and the “true” field contains an array with the values that are empty ([“”, null]). ![DataWeave Playground: groupBy isEmpty output grouping the array into false and true keys](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-1-sizeof-groupby-isempty-default-1.png) **Step 2**: Since we want to count the number of values that *are* empty, let’s go ahead and extract the “true” field. ```dataweave (["notEmpty", "", null] groupBy isEmpty($))."true" ``` ![DataWeave Playground: selecting the "true" key returns an array of the empty and null values](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-1-sizeof-groupby-isempty-default-2.png) **Step 3**: To count how many values are in this array ([“”, null]), let’s use the “sizeOf” function. ```dataweave sizeOf((["notEmpty", "", null] groupBy isEmpty($))."true") ``` ![DataWeave Playground: sizeOf returns 2 with a green type warning underline on the script](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-1-sizeof-groupby-isempty-default-3.png) Oh, what’s that green line? Ah…It turns out that DataWeave is assuming that any data type can be returned here. It’s not an error, but it’s a warning. You can get rid of it by explicitly coercing the data inside the “sizeOf” function into Array. ```dataweave sizeOf((["notEmpty", "", null] groupBy isEmpty($))."true" as Array) ``` > [!NOTE] > The “[sizeOf](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-sizeof)” function only accepts the Array, Object, Binary, and String data types as arguments. However, if you send a null value instead of an array, this will immediately fail. You can’t coerce Null to Array. ```dataweave sizeOf((null groupBy isEmpty($))."true" as Array) ``` What can we do now? Oh! What about the “default” operator? ```dataweave sizeOf((null groupBy isEmpty($))."true" default []) ``` ![DataWeave Playground: a null input with default [] returns 0 from sizeOf](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-1-sizeof-groupby-isempty-default-4.png) That works! Now let’s put back the array we had before, instead of the null value, and continue to the next step. ```dataweave sizeOf((["notEmpty", "", null] groupBy isEmpty($))."true" default []) ``` **Step 4**: We want to return a Boolean depending on whether there are empty values in the array or not. Let’s add a condition to return true when there *are* empty values or false if there aren’t any. ```dataweave if ( sizeOf((["notEmpty", "", null] groupBy isEmpty($))."true" default []) > 0 ) true else false ``` ![DataWeave Playground: an if/else on sizeOf greater than 0 returns the Boolean true](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-1-sizeof-groupby-isempty-default-5.png) We got what we wanted! Now we just have to make this a function and call it with different values to make sure that it works for all our cases. ## Setting up test values To create the function, we can simply define it over the 3 dashes (---) using the “fun” keyword. We can copy and paste our previous code for this new function, and replace the [hardcoded](https://en.wikipedia.org/wiki/Hard_coding) array with the function’s argument. Like this: ```dataweave %dw 2.0 output application/json fun containsEmptyValues(arr) = if ( sizeOf((arr groupBy isEmpty($))."true" default []) > 0 ) true else false --- ``` Now we can create the rest of the payload that we’ll be using to test. Let’s create an object that will contain some different examples of the data that the function will receive. Something like this: ```dataweave %dw 2.0 output application/json fun containsEmptyValues(arr) = if ( sizeOf((arr groupBy isEmpty($))."true" default []) > 0 ) true else false --- { nullValue: containsEmptyValues(null), emptyArray: containsEmptyValues([]), arrayWithEmptyString: containsEmptyValues([""]), arrayWithNull: containsEmptyValues([null]), arrayWithEmptyString2: containsEmptyValues(["1", ""]), arrayWithNull2: containsEmptyValues(["1", null]), arrayWithValues: containsEmptyValues(["1", "2"]), arrayWithEmptyObject: containsEmptyValues([{}]), arrayWithEmptyObject2: containsEmptyValues(["1",{}]), arrayWithNonEmptyObject: containsEmptyValues([{a:"b"}]) } ``` ![DataWeave Playground: the containsEmptyValues function tested against many cases returning true or false](../../assets/blog/how-to-check-for-empty-values-in-an-array-in-dataweave-part-1-sizeof-groupby-isempty-default-6.png) Note that there are 3 fields at the end to test the behavior of the function with Object. Even though this was not our initial use-case, our “containsEmptyValues” function works with the Object data type as well. Why? Because we’re using the “[isEmpty](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-isempty)” function, which accepts the Array, String, Null, and Object data types. **Quiz**: What happens when we change the “isEmpty” function to “[isBlank](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-isblank)” instead? *The answer will be revealed in the next post.* ## Does it work as expected? It works for *most* of our use-cases. However, it’s not the best solution. Here are some of the cons of this function: - It’s not very easy to read. There are a lot of parentheses that may confuse the developers that end up maintaining your code. - DataWeave is still showing a warning. Yes, the code runs just fine, and there are no errors, but warnings are there for a reason! - The null value (null) and the empty array ([]) are returning a false value, but they should be returning a true. This is a bug that needs to be fixed. Aaaand…We’re done with the first function! Remember that the following posts will explain more elegant solutions, so subscribe now to receive an email as soon as the next parts are published! Remember that the answer to the quiz, “What happens when we change the isEmpty function to isBlank instead?” will be revealed in the next post. Keep learning! -Alex --- ## Using Salesforce Search in Mule 4 Source: https://prostdev.com/post/using-salesforce-search-in-mule-4 | Published: Jan 26, 2021 | Category: Tutorials *GitHub repository with the Mule project can be found at the end of the post.* One of the most used actions when we work with Salesforce integrations is the use of query. It allows us to pull information from any table, do some subqueries, and pull relationship data. But there’s one action we, as developers, don’t use very often (sometimes we don’t even know this operation is available for us) and it is the Salesforce Object Search Language (**SOSL**). ## SOSL and SOQL Based on the [Force.com Developer documentation](https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql_sosl_intro.htm): A SOQL query is the equivalent of a SELECT SQL statement and searches the org database. SOSL is a programmatic way of performing a text-based search against the search index. Whether you use SOQL or SOSL depends on whether you know which objects or fields you want to search, plus other considerations. Use SOQL when you know which objects the data resides in, and you want to: - Retrieve data from a single object or from multiple objects that are related to one another. - Count the number of records that meet specified criteria. - Sort results as part of the query. - Retrieve data from number, date, or checkbox fields. Use SOSL when you don’t know which object or field the data resides in, and you want to: - Retrieve data for a specific term that you know exists within a field. Because SOSL can tokenize multiple terms within a field and build a search index from this, SOSL searches are faster and can return more relevant results. - Retrieve multiple objects and fields efficiently where the objects might or might not be related to one another. - Retrieve data for a particular division in an organization using the divisions feature. - Retrieve data that’s in Chinese, Japanese, Korean, or Thai. Morphological tokenization for CJKT terms helps ensure accurate results. Here are some differences between SOSL and SOQL. ![Image taken from Force.com Developer documentation](../../assets/blog/using-salesforce-search-in-mule-4-1.png) You might be asking yourself how this is helpful and how you can use it. Well, let’s think of a scenario. ## Scenario The general idea is to be able to process **User** information coming from any source and use the information to be able to validate if a **Contact** or **Lead** already exists in the platform using a specific external Id field. Based on the result we should be able to update a Contact / Lead or create a brand new Lead record. ![Flowchart deciding whether a user exists as a Contact or Lead, then updating or creating a Lead](../../assets/blog/using-salesforce-search-in-mule-4-2.png) ## Implementation I will create a pretty simple application to demonstrate how we can accomplish this. The Mule application would be created in Mule 4 and I will set a few records in a DataWeave component to simulate the input payload. ## input-data-flow ![input-data-flow: Scheduler, Logger, two Transform Message steps, and a Flow Reference](../../assets/blog/using-salesforce-search-in-mule-4-3.png) This flow contains a scheduler to manually trigger the integration for demonstration purposes. Then we set the incoming payload in a DataWeave component just like this: There’s a variable called “**originalPayload**”, which will be used to filter the information out once we get Salesforce information. In the next DW component (Preparing Search Request) we just convert all the external Id values from the original response to a plain string value concatenated by **OR**, making this an understandable payload value for the Salesforce search. The code looks like this: ```dataweave %dw 2.0 output application/java --- (payload map { ids: "\"" ++ ($.id) ++ "\"" }.ids) joinBy " OR " ``` ## salesforce-search-flow ![salesforce-search-flow: Salesforce Search then Transform Message steps mapping, grouping, and collecting results](../../assets/blog/using-salesforce-search-in-mule-4-4.png) This flow will be in charge of making the search call into Salesforce, grouping the response and creating the variables we need to filter the originalPayload with the existing records. In the Salesforce search, I will pass the next expression: ``` FIND { :ids } IN ALL FIELDS RETURNING Contact(Id,external_id__c,Email), Lead(Id,external_id__c,email) ``` Where the :ids parameter is the previous string we created separating the Ids by OR. In this search we are asking Salesforce to retrieve the records from Contacts and Leads searching in all the fields. After the information is returned, we can tell which fields we need from each object. ![Salesforce Search config with a SOSL FIND string returning Contact and Lead fields, ids set to payload](../../assets/blog/using-salesforce-search-in-mule-4-5.png) "Mapping Search Response" component just creates a map of the Salesforce results (payload.searchRecords). After this, we will group the information by type. We will use this script: ```dataweave %dw 2.0 output application/java --- (payload groupBy ((value, index) -> value."type")) ``` In the same component, I’m creating a variable called **salesforceResponseMap** which contains a key-value map we can access using a value to get the full record. ```dataweave %dw 2.0 output application/java --- { (payload map { (($.external_id__c):$) if $.Id != null }) } ``` “Collect by type” is a different variable that allows us to separate the records from the Contacts and Leads we found and set the Id as the main key in order to be able to filter the data in the next components. At this point we already know which Contacts and Leads have been found. ```dataweave %dw 2.0 output application/java --- { fromContacts: payload.Contact map (salesforceContact, IndexOfContact)->{ (id: salesforceContact.external_id__c) if (salesforceContact.external_id__c != null), }, fromLeads: payload.Lead map (salesforceLead, indexOfLeads)->{ (id: salesforceLead.external_id__c) if (salesforceLead.external_id__c != null), } } ``` ## filter-and-collecting-records ![filter-and-collecting-records flow with Transform Message steps to filter contacts and leads, then a Flow Reference](../../assets/blog/using-salesforce-search-in-mule-4-6.png) This flow will filter the data from the original payload by removing existing contacts from Salesforce and leaving the records that need to be created as Leads. “Filter Contacts / Update Contact” will take any existing records from the groupedObjects.fromContacts variable based on the Id using this script: ```dataweave %dw 2.0 output application/java --- vars.originalPayload filter (not (vars.groupedObjects.fromContacts.id contains ($.id))) ``` We are basically removing the records to an array from another one. In the same component we are doing basically the same but without the not operator so it means we are collecting the information that needs to be updated as Contact and we are able to map the fields we need to update. ```dataweave %dw 2.0 output application/java --- (vars.originalPayload filter ((vars.groupedObjects.fromContacts.id contains ($.id))) map (contact, indexOfContact) -> { Id: vars.salesforceResponseMap[contact.id].Id, FirstName: contact."First Name" }) ``` “Filter Leads / Update Leads” is basically the same but using the Leads group. Finally the remaining component collects the remaining information of records that need to be created as Leads in Salesforce and we can map the information. ## enqueue-batch-jobs The meaning of this job is just to set the payloads for update and create records, the only additional thing on this component is that we are specifying the **sObject** and externald **variables**, so, instead of adding a batch component for each type, dynamically we are dynamically passing the sObject for updates and sObject and externalId for upsert calls. This means we can reuse our batch processes. ![enqueue-batch-jobs flow alternating Transform Message and Flow Reference calls to update and upsert batches](../../assets/blog/using-salesforce-search-in-mule-4-7.png) Finally we can see the **batch processing flow**. One batch will focus in updating the objects and just control the response from Salesforce with a DW component like this: ```dataweave %dw 2.0 output application/json --- payload.items map { id: $.id, success: $.successful, (field: $.errors[0].fields[0]) if $.successful == false, (message: $.errors[0].message) if $.successful == false, (statusCode: $.errors[0].statusCode) if $.successful == false } ``` Basically, we can collect the responses and use them. ![update-salesforce-records-batch job: a Batch Step updating records, mapping the response, and logging](../../assets/blog/using-salesforce-search-in-mule-4-8.png) ![upsert-salesforce-records-batch job: a Batch Step upserting records, a Transform Message, and a Logger](../../assets/blog/using-salesforce-search-in-mule-4-9.png) There are some things to consider when using SOSL over SOQL: One of the advantages of this is that we are able to retrieve multiple objects in a call and we are saving a couple of API calls - This can be used on processes that need just a few records. Massive amounts of data might include some complexity on how we create the SOSL expression, but in the end we can just adjust the limits ([https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_sosl_limits.htm](https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_sosl_limits.htm)) in case we need to. Let me know if you think this is helpful and I will be happy to enhance this process as well. ## GitHub repository You can pull the code from this [repository](https://github.com/emoran/yucelmoran-salesforce-search) if you want to see the whole process working. --- ## DataWeave 2.0 core functions cheatsheet Source: https://prostdev.com/post/dataweave-2-0-core-functions-cheatsheet | Published: Jan 19, 2021 | Category: Guides This is a compilation of all the core functions that can be used in DataWeave 2.0 according to [MuleSoft's official documentation](https://docs.mulesoft.com/dataweave/latest/dw-core), separated by input and output. The link to each function's official documentation page is provided in the list. There you can find more details about the functions and examples of how to use them. You can use ctrl+F or cmd+F to search for specific keywords and get to the function you're looking for. > [!IMPORTANT] > This cheatsheet has been last updated for **DataWeave version 2.12** (Mule 4.12). ## Array to Array **Input:** Array | **Output:** Array - [Plus plus (++)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-plusplus) - Concatenates two values. - [Minus minus (--)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-minusminus) - Removes specified values from an input value. - [distinctBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-distinctby) - Iterates over the input and returns the unique elements in it. - [filter](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-filter) - Iterates over an array and applies an expression that returns matching values. - [find](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-find) - Returns indices of an input that match a specified value. - [flatMap](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-flatmap) - Iterates over each item in an array and flattens the results. - [flatten](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-flatten) - Turns a set of subarrays into a single, flattened array. - [map](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-map) - Iterates over items in an array and outputs the result into a new array. - [maxBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-maxby) - Iterates over the array and returns the highest value of [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) elements from it. - [minBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-minby) - Iterates over the array and returns the lowest array according to the provided expression. - [orderBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-orderby) - Reorders the array's elements based on the given criteria. - [reduce](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-reduce) - Applies a reduction expression to the elements in the array. - [unzip](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-unzip) - Groups the values of the given sub-arrays by matching indices (or indexes) and returns new sub-arrays with the matching indices. *Note: performs the opposite of zip.* - [zip](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-zip) - Merges elements from two arrays into an array of arrays. *Note: performs the opposite of unzip.* ## Array to Boolean **Input:** Array | **Output:** Boolean - [contains](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-contains) - Returns true if an input contains a given value, false if not. - [isEmpty](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-isempty) - Returns true if the given input value is empty, false if not. - [max](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-max) - Returns the highest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [maxBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-maxby) - Iterates over the array and returns the highest value of [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) elements from it. - [min](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-min) - Returns the lowest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [minBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-minby) - Iterates over the array and returns the lowest boolean according to the provided expression. - [reduce](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-reduce) - Applies a reduction expression to the elements in the array. ## Array to Date **Input:** Array | **Output:** Date - [max](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-max) - Returns the highest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [maxBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-maxby) - Iterates over the array and returns the highest value of [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) elements from it. - [min](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-min) - Returns the lowest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [minBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-minby) - Iterates over the array and returns the lowest date according to the provided expression. ## Array to DateTime **Input:** Array | **Output:** DateTime - [max](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-max) - Returns the highest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [maxBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-maxby) - Iterates over the array and returns the highest value of [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) elements from it. - [min](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-min) - Returns the lowest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [minBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-minby) - Iterates over the array and returns the lowest datetime according to the provided expression. ## Array to LocalDateTime **Input:** Array | **Output:** LocalDateTime - [max](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-max) - Returns the highest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [maxBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-maxby) - Iterates over the array and returns the highest value of [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) elements from it. - [min](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-min) - Returns the lowest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [minBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-minby) - Iterates over the array and returns the lowest localdatetime according to the provided expression. ## Array to LocalTime **Input:** Array | **Output:** LocalTime - [max](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-max) - Returns the highest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [maxBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-maxby) - Iterates over the array and returns the highest value of [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) elements from it. - [min](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-min) - Returns the lowest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [minBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-minby) - Iterates over the array and returns the lowest localtime according to the provided expression. ## Array to Number **Input:** Array | **Output:** Number - [avg](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-avg) - Returns the average of numbers listed in the array. - [indexOf](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-indexof) - Returns the index of the first occurrence of the specified element in this array, or -1 if this list does not contain the element. - [lastIndexOf](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-lastindexof) - Returns the index of the last occurrence of the specified element in a given array, or -1 if the array does not contain the element. - [max](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-max) - Returns the highest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [maxBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-maxby) - Iterates over the array and returns the highest value of [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) elements from it. - [min](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-min) - Returns the lowest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [minBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-minby) - Iterates over the array and returns the lowest number according to the provided expression. - [reduce](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-reduce) - Applies a reduction expression to the elements in the array. - [sizeOf](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-sizeof) - Returns the number of elements in an array. - [sum](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-sum) - Returns the sum of numeric values in the given array. ## Array to Object **Input:** Array | **Output:** Object - [groupBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-groupby) - Returns an object that groups items from an array based on specified criteria, such as an expression or matching selector. - [maxBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-maxby) - Iterates over the array and returns the highest value of [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) elements from it. - [minBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-minby) - Iterates over the array and returns the lowest object according to the provided expression. - [reduce](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-reduce) - Applies a reduction expression to the elements in the array. ## Array to String **Input:** Array | **Output:** String - [joinBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-joinby) - Merges an array into a string and uses the provided string as a separator between each item in the list. *Note: this is the opposite operation of splitBy.* - [max](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-max) - Returns the highest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [maxBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-maxby) - Iterates over the array and returns the highest value of [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) elements from it. - [min](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-min) - Returns the lowest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [minBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-minby) - Iterates over the array and returns the lowest string according to the provided expression. - [reduce](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-reduce) - Applies a reduction expression to the elements in the array. ## Array to Time **Input:** Array | **Output:** Time - [max](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-max) - Returns the highest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [maxBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-maxby) - Iterates over the array and returns the highest value of [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) elements from it. - [min](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-min) - Returns the lowest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [minBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-minby) - Iterates over the array and returns the lowest time according to the provided expression. ## Array to TimeZone **Input:** Array | **Output:** TimeZone - [max](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-max) - Returns the highest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [maxBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-maxby) - Iterates over the array and returns the highest value of [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) elements from it. - [min](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-min) - Returns the lowest [Comparable](https://docs.mulesoft.com/munit/latest/comparable-matchers-reference) value in the given array. - [minBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-minby) - Iterates over the array and returns the lowest timezone according to the provided expression. ## Array and String to String **Input:** Array, String | **Output:** String - [joinBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-joinby) - Merges an array into a string and uses the provided string as a separator between each item in the list. *Note: this is the opposite operation of splitBy.* ## Date to Boolean **Input:** Date | **Output:** Boolean - [isLeapYear](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-isleapyear) - Returns true if it receives a date for a leap year, false. if not. ## Date to Number **Input:** Date | **Output:** Number - [daysBetween](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-daysbetween) - Returns the number of days between two dates. ## Date and LocalTime to LocalDateTime **Input:** Date, LocalTime | **Output:** LocalDateTime - [Plus plus (++)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-plusplus) - Concatenates two values. ## Date and Time to DateTime **Input:** Date, Time | **Output:** DateTime - [Plus plus (++)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-plusplus) - Concatenates two values. ## Date and TimeZone to DateTime **Input:** Date, TimeZone | **Output:** DateTime - [Plus plus (++)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-plusplus) - Concatenates two values. ## DateTime to Boolean **Input:** DateTime | **Output:** Boolean - [isLeapYear](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-isleapyear) - Returns true if it receives a date for a leap year, false. if not. ## LocalTime and Date to LocalDateTime **Input:** LocalTime, Date | **Output:** LocalDateTime - [Plus plus (++)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-plusplus) - Concatenates two values. ## LocalTime and TimeZone to Time **Input:** LocalTime, TimeZone | **Output:** Time - [Plus plus (++)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-plusplus) - Concatenates two values. ## LocalDateTime to Boolean **Input:** LocalDateTime | **Output:** Boolean - [isLeapYear](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-isleapyear) - Returns true if it receives a date for a leap year, false. if not. ## LocalDateTime and TimeZone to DateTime **Input:** LocalDateTime, TimeZone | **Output:** DateTime - [Plus plus (++)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-plusplus) - Concatenates two values. ## Number to Array **Input:** Number | **Output:** Array - [to](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-to) - Returns a range with the given numbers. ## Number to Boolean **Input:** Number | **Output:** Boolean - [isDecimal](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-isdecimal) - Returns true if the given number contains a decimal, otherwise false. - [isEven](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-iseven) - Returns true if the number or numeric result of a mathematical operation is even, false if not. - [isInteger](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-isinteger) - Returns true if the given number is an integer, otherwise false. - [isOdd](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-isodd) - Returns true if the given number is odd, otherwise false. ## Number to Number **Input:** Number | **Output:** Number - [abs](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-abs) - Returns the absolute value of a number. - [ceil](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-ceil) - Rounds the number up to the nearest whole number. - [floor](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-floor) - Rounds a number down to the nearest whole number. - [mod](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-mod) - Returns the modulo (the remainder after dividing the dividend by the divisor). - [pow](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-pow) - Raises the value of the given base number to the given power. - [randomInt](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-randomint) - Returns a pseudo-random whole number from 0 to the given number. - [round](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-round) - Rounds the number up or down to the nearest number. - [sqrt](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-sqrt) - Returns the square root of the given number. ## Number to Range **Input:** Number | **Output:** Range - [to](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-to) - Returns a range with the given numbers. ## Object to Array **Input:** Object | **Output:** Array - [entriesOf](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-entriesof) - Returns an array of key-value pairs that describe the key, value, and any attributes in the input object. - [keysOf](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-keysof) - Returns an array of keys from key-value pairs within the input object. - [namesOf](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-namesof) - Returns an array of strings with the names of the keys from the given object. - [pluck](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-pluck) - Iterates over an object and returns an array of keys, values, or indices (or indexes) from the object. Useful for mapping an object into an array. *Note: similar to mapObject - mapObject returns an object and pluck returns an array.* - [valuesOf](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-valuesof) - Returns an array of the values from the given object's key/value pairs. ## Object to Boolean **Input:** Object | **Output:** Boolean - [isEmpty](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-isempty) - Returns true if the given input value is empty, false if not. ## Object to Number **Input:** Object | **Output:** Number - [sizeOf](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-sizeof) - Returns the number of key/value pairs in an object. ## Object to Object **Input:** Object | **Output:** Object - [Plus plus (++)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-plusplus) - Concatenates two values. - [Minus minus (--)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-minusminus) - Removes specified values from an input value. - [distinctBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-distinctby) - Iterates over the input and returns the unique elements in it. - [filterObject](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-filterobject) - Iterates a list of key-value pairs in an object and applies an expression that returns only matching objects, filtering out the rest from the output. - [groupBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-groupby) - Groups elements of an object based on criteria that the groupBy uses to iterate over elements in the input. - [mapObject](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-mapobject) - Iterates over the object using a mapper that acts on keys, values, or indices (or indexes) of that object. *Note: similar to pluck - mapObject returns an object and pluck returns an array.* - [orderBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-orderby) - Reorders the object's elements based on the given criteria. ## Object and Array to Object **Input:** Object, Array | **Output:** Object - [Minus minus (--)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-minusminus) - Removes specified values from an input value. ## String to Array **Input:** String | **Output:** Array - [find](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-find) - Returns indices of an input that match a specified value. - [splitBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-splitby) - Splits the given string into an array based on the given separating string. *Note: this is the opposite operation of joinBy.* ## String to Boolean **Input:** String | **Output:** Boolean - [contains](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-contains) - Returns true if an input contains a given value, false if not. - [endsWith](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-endswith) - Returns true if a string ends with a provided substring, false if not. - [evaluateCompatibilityFlag](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-evaluatecompatibilityflag) - Returns the value of the compatibility flag with the specified name. - [isBlank](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-isblank) - Returns true if the given string is empty, composed of whitespaces, or null; otherwise false. - [isDecimal](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-isdecimal) - Returns true if the given number contains a decimal, otherwise false. *Note: this function is designed for Number instead of String. You will receive a warning from DataWeave, or an error if the provided String contains something that can't be coerced into Number.* - [isEmpty](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-isempty) - Returns true if the given input value is empty, false if not. - [startsWith](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-startswith) - Returns true if the string starts with the given substring, otherwise false. ## String to Number **Input:** String | **Output:** Number - [indexOf](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-indexof) - Returns the index of the first occurrence of the specified string in this string, or -1 if this list does not contain the element. - [lastIndexOf](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-lastindexof) - Returns the index of the last occurrence of the specified element in a given array, or -1 if the array does not contain the element. - [sizeOf](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-sizeof) - Returns the number of characters in a string. ## String to Object **Input:** String | **Output:** Object - [groupBy](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-groupby) - Returns an object that groups characters from a string based on specified criteria, such as an expression or matching selector. ## String to String **Input:** String | **Output:** String - [Plus plus (++)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-plusplus) - Concatenates two values. - [filter](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-filter) - Iterates over a string and applies an expression that returns matching values. - [lower](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-lower) - Returns the given string in lowercase characters. - [replace (with)](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-replace) - Performs a string replacement from the given substring with the provided string. - [trim](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-trim) - Removes any blank spaces from the beginning and end of the given string. - [upper](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-upper) - Returns the given string in uppercase characters. ## String and Regex to Array **Input:** String, Regex | **Output:** Array - [find](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-find) - Returns indices of an input that match a specified value. - [match](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-match) - Matches the given regular expression from the given string and separates the results into capture groups inside an array. *Note: match will return a full match from the string. For several matches, use scan instead.* - [scan](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-scan) - Returns an array with all of the matches found in the given string, using the given regular expression. *Note: scan will return all of the matches from the string. For a full match, use match instead.* - [splitBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-splitby) - Splits the given string into an array based on the regular expression. *Note: this is the opposite operation of joinBy.* ## String and Regex to Boolean **Input:** String, Regex | **Output:** Boolean - [contains](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-contains) - Returns true if an input contains a given value, false if not. - [matches](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-matches) - Returns true if an expression matches the entire given string, otherwise false. ## String and Regex to String **Input:** String, Regex | **Output:** String - [replace (with)](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-replace) - Performs a string replacement from the substring found with the given regular expression. ## Time and Date to DateTime **Input:** Time, Date | **Output:** DateTime - [Plus plus (++)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-plusplus) - Concatenates two values. ## TimeZone and Date to DateTime **Input:** TimeZone, Date | **Output:** DateTime - [Plus plus (++)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-plusplus) - Concatenates two values. ## TimeZone and LocalDateTime to DateTime **Input:** TimeZone, LocalDateTime | **Output:** DateTime - [Plus plus (++)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-plusplus) - Concatenates two values. ## TimeZone and LocalTime to DateTime **Input:** TimeZone, LocalTime | **Output:** DateTime - [Plus plus (++)](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-plusplus) - Concatenates two values. ## Others Some of these next functions don't have any input parameters or don't return any output. Some accept more than two arguments. - [log](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-log) - Without changing the value of the input, log returns the input as a system log. So this makes it very simple to debug your code, because any expression or subexpression can be wrapped with log and the result will be printed out without modifying the result of the expression. The output is going to be printed in application/dw format. - [logDebug](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-logdebug) - Helper function that logs messages at Debug level. - [logError](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-logerror) - Helper function that logs messages at Error level. - [logInfo](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-loginfo) - Helper function that logs messages at Info level. - [logWarn](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-logwarn) - Helper function that logs messages at Warn level. - [logWith](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-logwith) - Without changing the value of the input, logWith returns the input as a system log at the specified level. - [match / case](https://docs.mulesoft.com/mule-runtime/4.3/dataweave-pattern-matching) - Pattern matching can be achieved by using a combination of match and case. Can be applied to literal values, expressions, data types, or regular expressions. - [now](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-now) - Returns current date and time in a DateTime. - [onNull](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-onnull) - Executes a callback function if the preceding expression returns a null value and then replaces the null value with the result of the callback. - [random](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-random) - Returns a pseudo-random number between 0.0 and 1.0. - [read](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-read) - Reads a string or binary and returns parsed content (for example: application/json, application/xml, application/csv, etc.). - [readUrl](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-readurl) - Reads a URL, including a classpath-based URL, and returns parsed content. *Note: similar to the read function.* - [then](https://docs.mulesoft.com/dataweave/latest/dw-core-functions-then) - Works as a pipe that passes the value returned from the preceding expression to the next (a callback) only if the value returned by the preceding expression is not null. - [typeOf](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-typeof) - Returns the type of the given value. - [uuid](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-uuid) - Returns a v4 UUID using random numbers as the source. - [with](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-with) - Helper function used with [replace](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-replace), [update](https://docs.mulesoft.com/mule-runtime/4.3/dw-values-functions-update), or [mask](https://docs.mulesoft.com/mule-runtime/4.2/dw-values-functions-mask). - [write](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-write) - Writes a value as a string or binary in a supported format. - [xsiType](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-xsitype) - Creates a xsi:type type attribute. *Note: useful for XML data types.* - Additional to the plus plus (++) function, you can also concatenate objects by using the Object Destructor {( )}. Learn more about this approach [here](https://www.prostdev.com/post/combining-objects-concatenation-in-dw-2-0) or learn how to use $( ) to concatenate strings [here](https://blogs.mulesoft.com/dev-guides/how-to-tutorials/review-concatenation-functions-dataweave/). ## Null-safe functions Here is a list of the functions that don't need an additional null-checker implemented. In other words, they accept null values and won't throw any error. - [distinctBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-distinctby) - [filter](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-filter) - [filterObject](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-filterobject) - [flatMap](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-flatmap) - [flatten](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-flatten) - [groupBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-groupby) - [isBlank](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-isblank) - [isEmpty](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-isempty) - [log](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-log) - [lower](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-lower) - [map](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-map) - [mapObject](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-mapobject) - [orderBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-orderby) - [pluck](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-pluck) - [trim](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-trim) - [typeOf](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-typeof) - [upper](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-upper) If you have any suggestions, comments, or noticed that something's wrong with the provided information; please reach out with a comment on this post or contact us [here](https://www.prostdev.com/contact). I hope this was useful for you! -Alex --- ## Understanding APIs (Part 6): What are HTTP Status Codes? Source: https://prostdev.com/post/understanding-apis-part-6-what-are-http-status-codes | Published: Jan 12, 2021 | Category: Guides We’re almost done learning the basic concepts we need to understand APIs better. Let’s recap what we have learned so far: - [Understanding APIs (Part 1): What is an API?](https://www.prostdev.com/post/understanding-apis-part-1-what-is-an-api) - We defined the initial diagram. - [Understanding APIs (Part 2): API Analogies and Examples](https://www.prostdev.com/post/understanding-apis-part-2-api-analogies-and-examples) - We talked about the Restaurant and Calculator analogies and defined the Human Resources API example. - [Understanding APIs (Part 3): What are HTTP Methods?](https://www.prostdev.com/post/understanding-apis-part-3-what-are-http-methods) - We learned the 5 most popular HTTP methods: GET, POST, PUT, PATCH, and DELETE. - [Understanding APIs (Part 4): What is a URI?](https://www.prostdev.com/post/understanding-apis-part-4-what-is-a-uri) - We learned that a URL is just a form of URI. We broke down a URI into protocol, host, and path. - [Understanding APIs (Part 5): Intro to Postman and Query Parameters](https://www.prostdev.com/post/understanding-apis-part-5-intro-to-postman-and-query-parameters) - We learned how to use Postman and what Query Parameters are. ![Diagram defined in Part 1, which contains the 4 aspects of the API + the Implementation.](../../assets/blog/understanding-apis-part-6-what-are-http-status-codes-1.png) The last 3 posts mentioned components sent in the Request that tell the API what data we want to receive. This time, we’ll be talking about a part of the Response: the HTTP Status Codes. ## What are HTTP Status Codes? HTTP Status Codes are numbers returned by the API which are included in the Response. They indicate if the operation you wanted to perform was a success or not, and they describe the kind of success/failure that happened. It’s basically a conversation with the server where it tells you, “here is the data you requested,” “your Request is not correct; please take a look,” “I’m having internet issues; please try again later.” There are a lot of Status Codes reserved for certain API Responses, but we will be looking at the most popular ones: 200, 201, 202, 204, 400, 401, 403, 404, and 500. You can find a complete list [here](https://www.restapitutorial.com/httpstatuscodes.html) with their definitions. > [!NOTE] > Any developer, API designer, or architect can choose the status codes they want the API to return. Some APIs only return 200, while others only return 200, 400, and 500. However, I will follow the best practices shown by [REST API Tutorial](https://www.restapitutorial.com/httpstatuscodes.html). **200 OK** This code is returned when the Request is a success, and there are no errors whatsoever. It’s mainly returned when using a GET or a POST. The Response will contain the data requested by the GET or the data created/updated by the POST. **201 Created** This code means that the Request is a success and that a new resource was created. For example, a new employee was added to the company list successfully. The Response usually contains the data that was just created. It’s mainly returned when using a POST or a PUT. **202 Accepted** This code is returned when some processing is still pending, but it hasn’t been completed yet. This code is especially useful when the back-end processing of the information will take too long to complete, so the user receives a confirmation of the Request along with a pointer to monitor the processing status. Note that this doesn’t mean that the Request was a success. It just means that the server received it, but it can still fail its processing. **204 No Content** This code means that the Request is a success, but there is no data to be returned. For example, when deleting a resource, nothing is returned because it no longer exists. It’s mainly returned when using DELETE. **400 Bad Request** The Request contains a syntax error that could not be understood by the server. For example, if your Request needs to have a field called “FullName” and you don’t send it to the API, it will return this error. **401 Unauthorized** This code is returned when the Request must contain some authorization information to use the API, like a username and a password. However, this data was not included or was wrong. **403 Forbidden** This code is returned when the API refuses to return some information. The Request and the authorization are ok, but for some other reason the data can’t be returned. For example, when you have a session opened for too long, it expires, and when you click a button or request additional info, you can’t anymore. **404 Not Found** This code is one of the most commonly seen from a web browser. It means the server didn’t find what you were looking for, mostly because it doesn’t exist. For example, if you try to access [http://google.com/test](http://google.com/test), you will get this error because the “test” path doesn’t exist. **500 Internal Server Error** This code is returned when there is an error inside the API. The Request was perfectly fine, and there were no errors when calling the server, but then something failed inside the Implementation. This is the most generic error that can be used when the server fails. ## HTTP Status Codes in Postman Go to Postman and send a new GET Request to “google.com”. You will receive an HTML response, and you will be able to see the HTTP Status Code right over the Response. > [!NOTE] > If you’re new to Postman, you should check out the previous post of this series: "[Understanding APIs (Part 5): Intro to Postman and Query Parameters](https://www.prostdev.com/post/understanding-apis-part-5-intro-to-postman-and-query-parameters)." ![Postman GET request to google.com showing the 200 OK status code highlighted above the response.](../../assets/blog/understanding-apis-part-6-what-are-http-status-codes-2.png) **Quiz**: What HTTP Status Code do you receive when sending a GET Request to “https://api.twitter.com/2/tweets/1234” from Postman? *The answer will be revealed in the next post.* ## Recap - **HTTP Status Codes** are numbers returned in the API’s Response to indicate if the Request was successfully processed or if there was an error. - **200 OK** means the Request is a success and returns the requested information. - **201 Created** means there is a new resource added, and the Response contains this new information. - **202 Accepted** means the Request is accepted but is still processing. - **204 No Content** means the Request is successful, but no data can be returned. - **400 Bad Request** means there is something wrong with your Request. - **401 Unauthorized** means you didn’t send the proper credentials to access the URI or perform the Operation. - **403 Forbidden** means the information can’t be returned for some reason, like a time limit. - **404 Not Found** means the resource/URI you are trying to access does not exist. - **500 Internal Server Error** means something went wrong inside the Implementation or within the server. I need to tell you this joke because it’s too good to miss. A parent tells the teenager to clean their room. The teenager responds, “202,” and the parent goes away. Some hours later, the parent comes back to see that the room is still dirty and asks why it isn’t clean yet. The teenager answers, “I said 202, not 200.” Sorry. [#GeekyHumor](https://www.prostdev.com/blog/hashtags/GeekyHumor) *Prost!* -Alex --- ## Why should an editor review your blog posts? Source: https://prostdev.com/post/why-should-an-editor-review-your-blog-posts | Published: Jan 5, 2021 | Category: Opinion Having a suitable format for your article (using bold, headers, subtitles, bullet points) is necessary so your readers don’t get lost in-between paragraphs. We already discussed this point in “Tip [#4](https://www.prostdev.com/blog/hashtags/4) - Develop the article” from a previous post - “[7 tips to start writing your technical blog post](https://www.prostdev.com/post/7-tips-to-start-writing-your-technical-blog-post).” But what about grammar? What about consistency? Sometimes we think we’re clear when we explain something because it makes sense in our heads. What if someone else reads the same text but understands it completely differently? This post will give you my perspective on why having editors review your content is the best way to bullet-proof your articles from possible misunderstandings. ## Create a review process for yourself If you want to create quality content for your readers, you should have a review process in place. For those of you who are software developers, you (should) probably have a Code Review process for when you create new code. You may create unit tests for your code or maybe some functional testing. Then your code is ready to be uploaded to a Pull Request, where a technical leader (or anyone else) will take a look at your code to make sure that you’re not making mistakes. Why shouldn’t you be the one approving your own Pull Request? Because you may miss things that other people might notice. Well, the same thing happens when you create content. It doesn’t have to be a long, bureaucratic process. You can at least use an external tool to correct the basic grammar, or you can share your document with another person before publishing to check that what you wrote makes sense to someone else. The best quality, though, comes from a professional editor. ## Using an external tool to review your grammar is not 100% accurate Using software to check your grammar doesn’t necessarily mean that the grammar will be perfect. Software makes mistakes! [Grammarly](https://www.grammarly.com/), [Google Docs](https://docs.google.com/), or [Microsoft Word](https://www.microsoft.com/en-us/microsoft-365/word) are great tools for editing, but they were programmed by humans (who also make mistakes). I normally create all my posts in Google Docs and double-check that my writing’s basic syntax is correct. If I get a warning or an error there, it’s *almost* always giving me the right suggestion. But remember, although these tools are great places to start editing, they aren’t foolproof. ## So, why should you consider consulting with a professional editor? External software may show you the misspelled words or the grammatically correct way to write a sentence. But it won’t tell you when you’re using a word too often or when a sentence “sounds weird” in a casual setting. Here are some of the advantages that I’ve experienced after working with editors for my content: - **Spelling.** Yes, the external software may get almost all of the misspelled words, but trust me, not all of them. - **Over-using commas.** It turns out that I use a lot of unnecessary commas when writing! None of those external tools have ever caught this. - **Misused punctuation.** I consider myself to have an excellent grasp on how to use almost all punctuation marks. Well, I definitely don’t know how to use *all* of them, and I get confused sometimes (*do you know the difference between an em dash and a hyphen without googling? ‘Cause I certainly don’t*). - **Keeping consistency.** Just the other day, I was writing a post and used “Google” every time I mentioned it…except once. Only one single time, I wrote “google,” and I didn’t notice (of course, my editor did!). Staying consistent throughout your article is important, so you don’t confuse your readers. - **Using synonyms.** Sometimes I think too fast, and I end up writing the same word 10 different times in the same paragraph. When you read that, it sounds way too repetitive! Well, your editor may know some synonyms to refer to the same thing without the paragraph sounding awkward. - **Re-writing sentences.** As a native Spanish speaker, my train of thought is normally in Spanish. There are some times when I *think* in English because I’m writing in English. But other times, I think first in Spanish, and then the translation is not as good as I had hoped for (like using Google Translate, LOL). Having an English-speaker editor helps a lot because the same sentence that makes sense to me (because it’s translated from Spanish) may sound weird to them, so they reformulate it. My documents sometimes end up with more than 50 suggestions/comments from my editors! And there I was, thinking that what I wrote was perfect and understandable to everyone. I hope this post was helpful to you. Of course, this is just my point of view on creating content, but I strongly recommend having someone else review your articles for higher quality posts. :) *Prost!* -Alex --- ## Implementing Pagination with MuleSoft Source: https://prostdev.com/post/implementing-pagination-with-mulesoft | Published: Dec 15, 2020 | Category: Tutorials ## Summary There could be a situation, in which we are consuming an API and the result is returned in multiple pages. Your business case needs the Mule Application to have a capability to process all results from all pages and return the response as per specification. Therefore, we are presented with a business problem to resolve – how to process n number of pages and return the combined response back to the calling client. As per design your backend API only returns x number of records per page and depending on the total number of records it could be n number of pages. So, as a consumer of that backend system and gateway to the client, you are responsible to maintain pagination. ## Solution Approach Let’s talk about how we resolve this problem effectively using MuleSoft. One simple solution to this problem is implementing a recursive flow that calls the backend system API n number of times until all records are fetched and no new pages exist. Pros: Main advantage of this approach is the simplicity and also that it’s easier to implement. Cons: - Synchronous approach. Hence, the next page cannot be fetched until the processing of prior pages are completed. - Response would be slow. - Main disadvantage is, if we have many records and if you need to iterate too many times, Mule Runtime might crash with “Too many child contexts” error. An alternative solution to this problem is to implement this in a more complex way and asynchronous processing could be a saviour here. To demonstrate this approach, we will be calling the GitHub API to return all the repositories under a specific user/organization. There are a few steps of authentication before we can call the GitHub API to get all the results. ## Authentication for GitHub API There are two ways to authenticate through the GitHub REST API. 1) Basic authentication ```bash $ curl -u "username" https://api.github.com ``` 2) OAuth2 token (sent in a header) ```bash $ curl -H "Authorization: token OAUTH-TOKEN" https://api.github.com ``` The recommended way of authentication is OAuth tokens using Authorization Header. As a part of this implementation, we will be doing authorization as OAuth using a Personal Access Token. Personal Access Tokens can be generated for each user and can be sent as Authorization header. If you want to know more details about how to create a Personal Access Token for your GitHub account, you can follow [this link](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token). ## Implementation Main implementation involves two steps: Step 1: Initial step is to get the results for the first page. This is handled by the GetFirstPageResponse sub-flow which essentially calls the GitHub API with page number as 1. Step 2: Once we have page 1 result, we will call ProcessPageResults sub-flow to fetch subsequent pages and appending the results of all pages. ## Step 1: Getting First Page from GitHub API Getting the first page is straight forward. We set up essential variables like pageNumber which will facilitate calling the GitHub API. Then, we invoke the GitHub API to get the results of the first page. For more details about how to call the GitHub API, you can follow the section “Invoking GitHub API.” Next step is storing this result in an Object Store. We store the results of first page in Object Store to smoothly append the results of all pages at the end of the process which we will be seeing in a moment. ![GetFirstPageResponse sub-flow setting pageToBeVisited, calling InvokeGithubAPI and storing the result](../../assets/blog/implementing-pagination-with-mulesoft-1.png) ## Step 2: Processing Subsequent Pages From the previous section, we have fetched the first page of GitHub API, now it’s our turn to process the results from other pages. In this section, we will be discussing about processing of subsequent pages asynchronously. ![ProcessPageResults sub-flow with a For Each that stores status, sets VM payload and publishes](../../assets/blog/implementing-pagination-with-mulesoft-2.png) Once we have the first page, we will examine the response headers. Usually, in the response header, GitHub indicates the information about pagination. GitHub indicates the following information in the header called **link**. - **Prev**: contains the link to the previous page. - **Next**: contains the link to the next page. - **First**: contains the link to the first page. - **Last**: contains the link to the last page. Our concern is all about the last page link information. We will be extracting page numbers from this link type. That would indicate how many more pages we still need to process. For example: If the page number of the last link says 35, then we have 35 - 1 = 34 pages to be processed (since the first page has already been processed). After getting the information about how many more pages we still have to process, we can make up an array of page numbers which are still required to process. ```dataweave %dw 2.0 output application/json var linkHeader = attributes.headers.link var linkHeaderArr = (linkHeader splitBy ",") fun createLinkHeaderDetails() = { linkHeaderDetails: linkHeaderArr map ( createLinkHeaderObject($) ) } fun createLinkHeaderObject(linkHeaderObj) = { "pageNumber": (((((((linkHeaderObj splitBy ";") map (trim($)))[0]) splitBy "&")[1]) splitBy "=")[1])[0], "linkType": readString(((((linkHeaderObj splitBy ";") map (trim($)))[1]) splitBy "=")[1]) } fun readString(readString) = read(readString,"application/json") fun getLastPageNumber() = createLinkHeaderDetails().linkHeaderDetails filter ($.linkType == "last") --- (1 to (getLastPageNumber().pageNumber[0] - 1)) map { pageNumber: ($ + 1) } ``` Next step is: for each page number in a page number array, we are setting up the following: - We will store a flag in Object Store for each specific page number that indicates the processing status. Initially it will be false. - We will be setting up a payload that contains the page number we want to fetch. This is done using a simple DataWeave script to set up the payload for the VM endpoint flow. ```dataweave %dw 2.0 output application/java --- { pageNumber: payload.pageNumber } ``` Then, we will be publishing this payload to a VM endpoint which is exposed by another flow that essentially fetches the results from the GitHub API. This will be discussed in the section “Retrieve Details from GitHub API.” This approach makes this process asynchronous since it won’t wait for the VM endpoint to come back with a response. The VM Publish endpoint works like Fire and Forget, so, the Mule thread won’t wait for this process to be completed. That’s why we call another sub flow named WaitUntilAllPagesAreProcessed outside of the For Each scope which is responsible for checking whether all pages are completely processed or not. This flow is not going to end until all pages are processed. ![WaitUntilAllPagesAreProcessed sub-flow with a Choice branching on whether all pages are processed](../../assets/blog/implementing-pagination-with-mulesoft-3.png) The process to know if all pages have been processed is determined by checking the entry in Object Store for each page number that we set before publishing to the VM endpoint. If the page results are processed, this entry is supposed to be updated to true in Object Store which is part of discussion in the “Retrieve Details from GitHub API” section. We can check for the Object Store entries for all the pages with a simple DataWeave script. ```dataweave %dw 2.0 output application/json fun checkForCompletion(payload) = not sizeOf ((payload pluck $) default [] filter ($ == false)) >0 --- checkForCompletion(payload) ``` If the processing flag of any page is still false, we will impose a small delay with a DataWeave script and then call WaitUntilAllPagesAreProcessed recursively until all pages processing is completed. This delay is imposed to avoid having Mule to crash because of too many recursive calls to the flow. This delay can be configured from configuration parameters and we will keep it to a minimum number (i.e., 1000 ms) which will allow the processing of the pages to be completed and avoid too many calls to the same flow. If the processing of all pages is complete, we fetch the final response from Object Store entry and transform it as per our requirements to form a final response to client. ## Retrieve Details from GitHub API In this section we will be talking more about the flow which is responsible for fetching details from GitHub API and processing the results. We will be building this flow in a way so that it can be called asynchronously. One way to achieve this is to implement a VM Listener endpoint so that we can call this flow asynchronously from other flows. For this flow to work, we only need the page number (which needs to be fetched) to be passed as a payload. ![GetRepoDetailsAsync flow with a VM Listener setting variables and calling InvokeGithubAPI and Append flows](../../assets/blog/implementing-pagination-with-mulesoft-4.png) Let’s discuss this flow in couple of steps: Step 1: We will be retrieving details from the Object Store regarding the GitHub API call, like page size, which is required to tell the GitHub API how many records it’s going to send as a response for the page. Step 2: We will be setting up two variables named pageToBeVisited and pageSize. First variable will get the value from the payload that was passed to the VM endpoint which indicates the specific page number that we are going to fetch while later suggests the number of records for that page to be returned which we can get from Step 1. Step 3: We will call the InvokeGitHubAPI flow that essentially handles the complexity of calling the GitHub API and returning the results. This will be discussed in the following section, “Invoking GitHub API.” Step 4: Next step is appending the result of the current page with previous pages. That can be done by retrieving the result from the Object Store key entry (which is initially created when we fetched the result of Page 1, appending the current page result and storing it back to the same Object Store key. ![AppendGithubResults sub-flow with a Try retrieving the response, then appending and storing it](../../assets/blog/implementing-pagination-with-mulesoft-5.png) Step 5: Finally, we need to update the processing status to true in the Object Store key created for that specific page (discussed in “Step 2: Processing Subsequent Pages"). This indicates that the processing for this specific page is completed. ![UpdateProcessingStatus sub-flow with a Try retrieving the entry then storing the updated status](../../assets/blog/implementing-pagination-with-mulesoft-6.png) ## Invoking GitHub API GitHub REST API offers many functionalities - from getting details of repositories to push/pull the commits. You can read more about the GitHub API here to know what it has to offer. At the time of the writing this article, GitHub offered v3 of its implementation. By default, all requests to [https://api.github.com](https://api.github.com/) receive the v3 [version](https://docs.github.com/en/free-pro-team@latest/developers/overview/about-githubs-apis) of the REST API. However, you can also request a previous version if you need. For our implementation, we will be using the latest version. All API access is over HTTPS and accessed from [https://api.github.com](https://api.github.com/). All data is sent and received as JSON. There are many resource endpoints available but for this implementation, we will only be focusing on GET /user/repos resource to get repositories that the authenticated user has explicit permission (:read, :write, or :admin) to access. The authenticated user can get information on repositories they own, repositories where they are a collaborator, and repositories that they can access through an organization membership. Since this endpoint accepts GET HTTP method, it does not need request payload rather it accepts few query parameters and response comes in application/json format. You will find more details about this endpoint [here](https://docs.github.com/en/free-pro-team@latest/rest/reference/repos#list-repositories-for-the-authenticated-user). To implement this in Mule, we will be building a flow in the following steps: ![InvokeGithubAPI sub-flow setting the authToken variable and making an HTTP Request to GitHub](../../assets/blog/implementing-pagination-with-mulesoft-7.png) Step 1: We setup a variable named authToken that gets the encrypted token from configuration file appended by string “token”. For more details about how to obtain this token, please see section “Authentication for GitHub API.” Step 2: Next step is making a HTTP request to the GitHub API endpoint we discussed earlier. There are a couple of headers and query parameters which are required. **Headers**: Authorization: This header will be used to authenticate the user with the GitHub API. We will be using the value from variable authToken that we created in Step 1. There are number of ways to authorize any user via GitHub API to access protected resources. For this implementation, we have used OAuth with Personal Access Token. The details to implement this in GitHub can be found in section “Authentication for GitHub API.” **Query Parameters**: per_page: This query parameter indicates how many records we want to fetch per page. Default is 30 records per page. However, for some endpoints, we can go up to 100 records per page. If you have more items than the ones you have requested using this query parameter, your results will be paginated. For more information about how pagination works with GitHub API, you can click [here](https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#pagination). page: Page number that needs to fetch. ## Conclusion Thank you for taking the time to read this article. We have learnt how to effectively and efficiently retrieve large datasets by iterating over a number of pages returned from the backend API. This implementation solely focuses on the GitHub REST API for demonstration, but I believe it can be tailored to your technical needs. Here are few references that may help you to get started- [https://docs.github.com/en/free-pro-team@latest/rest/reference](https://docs.github.com/en/free-pro-team@latest/rest/reference) [https://docs.mulesoft.com/vm-connector/2.0/](https://docs.mulesoft.com/vm-connector/2.0/) [https://docs.mulesoft.com/vm-connector/2.0/vm-examples](https://docs.mulesoft.com/vm-connector/2.0/vm-examples) --- ## Understanding APIs (Part 5): Intro to Postman and Query Parameters Source: https://prostdev.com/post/understanding-apis-part-5-intro-to-postman-and-query-parameters | Published: Dec 8, 2020 | Category: Tutorials Part 5 already! What a trip it’s been. :) Let’s recap what we have learned so far: - [Understanding APIs (Part 1): What is an API?](https://www.prostdev.com/post/understanding-apis-part-1-what-is-an-api) - We defined the initial diagram. - [Understanding APIs (Part 2): API Analogies and Examples](https://www.prostdev.com/post/understanding-apis-part-2-api-analogies-and-examples) - We talked about the Restaurant and Calculator analogies and looked at the Human Resources API example. - [Understanding APIs (Part 3): What are HTTP Methods?](https://www.prostdev.com/post/understanding-apis-part-3-what-are-http-methods) - We learned the 5 most popular HTTP methods: GET, POST, PUT, PATCH, and DELETE. - [Understanding APIs (Part 4): What is a URI?](https://www.prostdev.com/post/understanding-apis-part-4-what-is-a-uri) - We learned that a URL is just a form of URI. We broke down a URI into protocol (HTTP/HTTPS), host, and path. ![Diagram defined in Part 1, which contains the 4 aspects of the API + the Implementation.](../../assets/blog/understanding-apis-part-5-intro-to-postman-and-query-parameters-1.png) Continuing with the last topic, URIs, we will now get started using Postman and learning what Query Parameters are. ## Getting started with Postman You can download Postman for free from their [website](https://www.postman.com/downloads/). Once you have it, open it, and it will look something like this: ![Postman app home screen with an empty Collections sidebar and the workspace launchpad](../../assets/blog/understanding-apis-part-5-intro-to-postman-and-query-parameters-2.png) Now let’s click on the plus (+) button as indicated below. ![Postman toolbar with the plus button for a new request highlighted by a red box](../../assets/blog/understanding-apis-part-5-intro-to-postman-and-query-parameters-3.png) This will open a new Untitled Request. Let’s paste the following URI into the text box that says “Enter request URL” - ``` https://official-joke-api.appspot.com/random_joke ``` Now click on “Send.” You should be able to see a formatted Response similar to this: ![Postman GET request to the Joke API showing the HTTP method, URI, and JSON response labeled](../../assets/blog/understanding-apis-part-5-intro-to-postman-and-query-parameters-4.png) > [!NOTE] > If the Response appears at the bottom of the screen and not on the right, you can change Postman’s layout by going to Postman preferences (or settings) > General tab > Check that the User Interface is set as Two-pane view. Congratulations! You just made a call to the [Joke API](https://github.com/15Dkatz/official_joke_api) using Postman. ## What are Query Parameters? Query Parameters (also known as Query Params), which are appended to the URI, are fields that are used to send additional information or data to the API. They are usually used to filter or search for data. For example, if you want to search for “cats” from Google, you can simply use this URI: [https://www.google.com/search?q=cats](https://www.google.com/search?q=cats). Let’s break down this URL like we did in the last post. ![Table breaking the Google search URL into protocol, host, path /search, and query param q=cats](../../assets/blog/understanding-apis-part-5-intro-to-postman-and-query-parameters-5.png) In this case, there’s only one Query Param, which is named “q,” and the value we’re sending is “cats.” For Google, when you call the path “/search” and attach the “q” Query Parameter, it knows that you want to perform a Google search with the word “cats.” So what are the elements of a Query Param? - **(?)** - We can identify the Query Parameters in a URI because they follow the question mark (?). - **(=)** - The Query Param name will be located before the equal sign (=), and the value will go after it. In this case, the value “cats” is being assigned to the Query Param called “q.” - **(&)** - If there were more Query Params in the same URI, they would be attached using the ampersand character (&). E.g. ``` myhost.com/mypath?queryparam1=cats&queryparam2=dogs&queryparam3=squirrels ``` ## Example: Expedia Let’s now get into a more complex example. In your browser, go to [Expedia.com](https://www.expedia.com/) and search for a hotel. ![Expedia search form with Toronto as destination and Nov 23–24 check-in and check-out dates](../../assets/blog/understanding-apis-part-5-intro-to-postman-and-query-parameters-6.png) After you click “Search,” take a look at the URL in your browser. ![Browser address bar showing the long Expedia Hotel-Search URL full of encoded query parameters](../../assets/blog/understanding-apis-part-5-intro-to-postman-and-query-parameters-7.png) For me, it looks like this: ``` https://www.expedia.com/Hotel-Search?adults=2&d1=2020-11-23&d2=2020-11-24&destination=Toronto%20%28and%20vicinity%29%2C%20Ontario%2C%20Canada&endDate=2020-11-24®ionId=178314&rooms=1&semdtl=&sort=RECOMMENDED&startDate=2020-11-23&theme=&useRewards=false&userIntent ``` Let’s put this URI in Postman and see what happens. ![Postman Query Params table listing the Expedia parameters such as adults, dates, and destination](../../assets/blog/understanding-apis-part-5-intro-to-postman-and-query-parameters-8.png) A bunch of Query Params just appeared! Postman is smart enough to know that everything that comes after the question mark (?) are Query Parameters. They are shown in this neat way to read/modify/add/delete them more easily. If you click “Send,” you will see some of the HTML code behind the Expedia website. This is the data that your browser sees and then translates into visuals for you. ![Postman showing the Expedia response body as raw HTML markup after sending the request](../../assets/blog/understanding-apis-part-5-intro-to-postman-and-query-parameters-9.png) Take a look at the “destination” Query Param. Its value will look something like this: ``` Toronto%20%28and%20vicinity%29%2C%20Ontario%2C%20Canada ``` Notice how there are some numbers after the percentages (%) like “%20”, “%28”, “%29”. This is because the URIs can’t contain some characters such as a space, parentheses or commas. So such characters are translated into these “%20” encodings. Here are some examples: ![W3Schools.com - HTML URL Encoding Reference.](../../assets/blog/understanding-apis-part-5-intro-to-postman-and-query-parameters-10.png) So, if we translate “Toronto%20%28and%20vicinity%29%2C%20Ontario%2C%20Canada” back into the original string, this is what it says: “Toronto (and vicinity), Ontario, Canada”. What would happen if we paste the original string directly into the “destination” Query Param in Postman? Let’s do it and resend the Request to see if it works. ![Postman destination param set to the plain text "Toronto (and vicinity), Ontario, Canada" working fine](../../assets/blog/understanding-apis-part-5-intro-to-postman-and-query-parameters-11.png) It worked! Postman is smart enough to know that these characters will need to be translated into the ASCII Encoding and does it in the background without us noticing. If you try to call this URL from your browser, it will also work, but your browser will show you the translated URL just as before. Notice how all these Query Params are information that the backend will use to filter the results that it will show you. For example, adults=2, rooms=1, endDate=2020-11-24, startDate=2020-11-23. With these Params, the server knows not to show you hotels that won’t be available for two adults or that won’t be available from November 23 to November 24. ## Recap - Postman is a tool that can help us call our APIs with a user-friendly interface. You can download the application from their website. - Postman can automatically format the JSON Responses you get from an API, so it’s more human-readable. - Postman shows your Query Params so you can conveniently add/read/modify/delete them as you wish. - Query Parameters (also known as Query Params), which are appended to the URI, are fields that are used to send additional information or data to the API. - There is a special HTML URL Encoding that needs to be used to read special characters like spaces, commas, or parentheses. Postman translates this automatically for you. Even though we didn’t use the Expedia API, we were able to call the Expedia website. We learned how the Query Parameters work in a real-life example and how to identify them in a URI. In the next post, we will be learning about a component that is found in the Response of our APIs: the HTTP Status Codes. Subscribe now to receive a notification as soon as this post is published! *Prost!* -Alex --- ## Anypoint Platform Chrome Extension Source: https://prostdev.com/post/anypoint-platform-chrome-extension | Published: Dec 1, 2020 | Category: Guides As developers we always want to have information at first hand whenever we need to access information. One of the main reasons this extension was developed is that currently whenever we access Anypoint Platform we need to navigate with many clicks in the portal to see information about the organization or about a specific application. For example, what alerts does this application have? Or, does this application have any schedulers? My motivation for developing this extension is to know what the MuleSoft community is looking for with tools like this and to know how we can create something useful for everybody. We started really basic on this one but definitely the idea would be to make it as robust and helpful as we can. In this post, I’ll show you some of the features that you can find in the Anypoint Platform Chrome Extension so far. ## See Organization name, Organization id and Environment id These parameters are important for you, as a developer. Whenever you need to make an HTTP call using the CloudHub or Hybrid API, or when you are about to use Anypoint CLI. This information is visible at the top of the extension. ![Chrome extension header showing the Organization name, Organization id, and Environment id.](../../assets/blog/anypoint-platform-chrome-extension-1.png) ## This only works so far on the Runtime Manager page There is still a lot of work to do. For now, the extension only works on the Runtime Manager page and diving into the Runtime Manager applications. I will try to add some information for the main pages. *Please don't judge so bad. :P* ## Runtime Manager Application Once you select a specific application on Runtime Manager, you should be able to see more information in the extension. For example, on the top you should be able to see the name of the application, the target system where it’s deployed, and its current status. ![Extension view of an app named configuration-mapper, target CloudHub, status Started, with a Download Artifact button.](../../assets/blog/anypoint-platform-chrome-extension-2.png) As you can see above, there is a button allowing you to download the artifact (directly from your browser). If the application deployed is a Mule 4 application, you will get a .jar file. If the application was deployed in Mule 3.9.X, the artifact will be downloaded as a .zip file. ## See schedulers created in the application Right now to be able to see if there are schedule processes in our application, we need to go into the application, click on “schedulers,” and there we are able to enable or disable the schedulers. In this case, in the application (and using the extension) you should be able to see the schedulers created in CloudHub and Hybrid. ![Image from a CloudHub application](../../assets/blog/anypoint-platform-chrome-extension-3.png) ![Hybrid application view](../../assets/blog/anypoint-platform-chrome-extension-4.png) ## See alerts related to the application You are able to see related alerts in the application just by clicking on the "Alerts" tab. The good thing about this is that you’re able to click on the alert and be redirected to make the changes that you need. ![Extension Alerts tab listing a critical CPU alert that is enabled for the application.](../../assets/blog/anypoint-platform-chrome-extension-5.png) Finally, you can get the extension from [this link](https://chrome.google.com/webstore/detail/anypoint-platform-chrome/gofndnflkobgljnjjalmehnlamoifmhc?hl=en&authuser=0) (there are a few bug fixes on version 1.5, so wait for it). Feel free to use the support page on the Chrome Extension page to give me any feedback, I'm really looking forward to hearing from the community about this. Link to the original post can be found here: [Anypoint Platform Chrome Extension at yucelmoran.com](https://yucelmoran.com/anypoint-platform-chrome-extension/). --- ## Understanding APIs (Part 4): What is a URI? Source: https://prostdev.com/post/understanding-apis-part-4-what-is-a-uri | Published: Nov 24, 2020 | Category: Guides We are slowly starting to get a sense of what an API is. So far, we’ve been learning, piece by piece, some of the components that define an API. Let’s take a look back at what we’ve learned: - [Understanding APIs (Part 1): What is an API?](https://www.prostdev.com/post/understanding-apis-part-1-what-is-an-api) - We defined the 4 aspects: Inputs, Operations, Outputs, and Data Types. We also explained what an API’s Request and Response is. Finally, we put all this together in a diagram. - [Understanding APIs (Part 2): API Analogies and Examples](https://www.prostdev.com/post/understanding-apis-part-2-api-analogies-and-examples) - To get a better grasp, we talked about the Restaurant and Calculator analogies. Then, we defined the Human Resources API example using the CSV and JSON Data Types. - [Understanding APIs (Part 3): What are HTTP Methods?](https://www.prostdev.com/post/understanding-apis-part-3-what-are-http-methods) - We started getting technical and learned about the 5 most popular HTTP methods used by RESTful APIs: GET, POST, PUT, PATCH, and DELETE. ![API diagram: Request input with data type and operation, flowing through Implementation to a Response output](../../assets/blog/understanding-apis-part-4-what-is-a-uri-1.png) *Diagram defined in Part 1, which contains the 4 aspects of the API + the Implementation.* For this post, we will learn what a URL and a URI really are and how this fits in our previously defined diagram. ## What is a URL? In the previous post, we learned that RESTful APIs or RESTful Web Services are calls that are made through [HTTP](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) (Hypertext Transfer Protocol). When you’re using a telephone to make a call, you need to know the telephone number to dial. When you want to make a call using HTTP, you need to know the web address instead of the telephone number. This web address is known as a [URL](https://en.wikipedia.org/wiki/URL#:~:text=A%20typical%20URL%20could%20have,and%20a%20file%20name%20(%20index.) (Uniform Resource Locator). E.g. [https://www.google.com](https://www.google.com/). URLs are mostly used to retrieve websites or web pages from the internet (using [HTTP](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol)) such as Google or Facebook. But URLs can also be used for File Transfer (through [FTP](https://en.wikipedia.org/wiki/File_Transfer_Protocol)), database access (using [JDBC](https://en.wikipedia.org/wiki/Java_Database_Connectivity)), and many others. ## What is a URI? From Wikipedia: “The most common form of URI is the Uniform Resource Locator (URL), frequently referred to informally as a web address.” Actually, a [URL](https://en.wikipedia.org/wiki/URL#:~:text=A%20typical%20URL%20could%20have,and%20a%20file%20name%20(%20index.) is just a form of [URI](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier) (Uniform Resource Identifier). A lot of people use both terms to refer to the same thing in the API world. For the sake of this post, let’s just say that a URL is mostly used for visual websites, and we’ll be using URIs to call our APIs. E.g. [https://api.twitter.com/2/tweets/:id](https://api.twitter.com/2/tweets/:id). ## Example: Joke API I went ahead and googled some public APIs that I could use as examples, and I found this [Joke API](https://github.com/15Dkatz/official_joke_api) that, when called, returns with a joke in JSON format. ![Official Joke API documentation listing endpoints for random jokes, ten jokes, and jokes by type](../../assets/blog/understanding-apis-part-4-what-is-a-uri-2.png) *Official Joke API documentation is taken from the* [README.md](https://github.com/15Dkatz/official_joke_api/blob/master/README.md) *file.* Since these are all using GET as the HTTP method, you can open these links directly from your browser (or just click on them). Let’s check out the first option, which is to grab a random joke: [https://official-joke-api.appspot.com/random_joke](https://official-joke-api.appspot.com/random_joke) This is what I get in my browser after clicking or inputting the link: ```json {"id":131,"type":"general","setup":"How do you organize a space party?","punchline":"You planet."} ``` It may be a bit hard to read if you’re not familiar with the JSON syntax, but it basically returned a setup that says, “How do you organize a space party?” and the punchline is “You planet.” Let’s try now to retrieve a random programming joke: [https://official-joke-api.appspot.com/jokes/programming/random](https://official-joke-api.appspot.com/jokes/programming/random) Now I got this as the Response: ```json [{"id":377,"type":"programming","setup":"Knock-knock.","punchline":"A race condition. Who is there?"}] ``` Let me show you a trick to be able to format a JSON into something more human-readable. Go to [this JSON formatter website](https://jsonformatter.curiousconcept.com/) and paste your JSON data into the text box. ![JSON Formatter & Validator website with the raw programming-joke JSON pasted, ready to Process](../../assets/blog/understanding-apis-part-4-what-is-a-uri-3.png) Then just click on “Process,” and you will get something like this: ![JSON formatter showing the indented, human-readable programming joke marked valid per RFC 8259](../../assets/blog/understanding-apis-part-4-what-is-a-uri-4.png) Isn’t it more human-readable now? P.S. I didn’t get the joke. :( ## Breaking down a URI Let’s break down the two URIs we just called: the random joke and the random programming joke. **Random joke** ![Table breaking the random_joke URI into protocol HTTPS, host, and path /random_joke](../../assets/blog/understanding-apis-part-4-what-is-a-uri-5.png) **Random programming joke** ![Table breaking the URI into protocol HTTPS, host, and path /jokes/programming/random](../../assets/blog/understanding-apis-part-4-what-is-a-uri-6.png) As we can see here, the HTTP or HTTPS protocols will be used when using RESTful APIs. HTTPS is just a secure way to call HTTP. The host is the URI part that remains the same even when you change the path you’re using to retrieve different information. Just like how “google.com” doesn’t change even after searching for something. Those two were kind of straightforward to break down, but in the following post, we will learn about Query Parameters and start using a great tool called Postman. With it, you will be able to call your APIs and see the responses in a more human-readable way without using the browser or the JSON formatter tool. ## Recap - A **URL** (Uniform Resource Locator) is the web address that you use in your browser to retrieve a website. It’s also a form of URI. - A **URI** (Uniform Resource Identifier) is what we will be referring to when we call our API. - An API’s URI uses **HTTP** (Hypertext Transfer Protocol) or **HTTPS** (Hypertext Transfer Protocol Secure). - The **host** is the part of the URI that remains the same even when you change the path you’re using to retrieve information. APIs are slowly taking form for us. We already called an API from our browser today! Stay tuned to learn more about Postman and more exciting real-life examples of APIs. *Prost!* -Alex --- ## Understanding APIs (Part 3): What are HTTP Methods? Source: https://prostdev.com/post/understanding-apis-part-3-what-are-http-methods | Published: Nov 3, 2020 | Category: Guides In [part 1](https://www.prostdev.com/post/understanding-apis-part-1-what-is-an-api) of this series, we learned what an API is and the 4 aspects that we can use to define it, which are Inputs, Operations, Outputs, and Data Types. We also outlined this nice diagram to have a visual representation: ![Diagram of an API: Request input plus operation, an Implementation box, and a Response output](../../assets/blog/understanding-apis-part-3-what-are-http-methods-1.png) In the [previous post (part 2)](https://www.prostdev.com/post/understanding-apis-part-2-api-analogies-and-examples), we described some analogies and examples to understand better how APIs can be applied in the real world. There was also a quiz for the second analogy (the calculator). The question was: *What would be the Data Types for the RQ and RS in this scenario?* The answer will be revealed throughout this post! So far, we’ve been talking about the theory behind APIs, but we have to get our hands dirty at some point! Today we will be learning more about these HTTP methods and how they fit into our previously defined diagram. ## What exactly is an API? API can mean a lot of things in the software development world, but I will be talking about the APIs that have been popular lately: the RESTful APIs or RESTful Web Services. These APIs are “calls” (the combination of Request and Response sent to and received from the API) that are made through [HTTP](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) (Hypertext Transfer Protocol). To explain HTTP more simply, when you type “www.google.com” or “facebook.com” into your browser, and you press Enter, you’re communicating with a remote server via HTTP. You are sending a Request to a specific web address, and you are getting back a Response, which your browser shows visually (Google or Facebook’s website, in this case). ## Understanding the Operation One of the 4 aspects that we previously defined was the Operation that is sent to the API in the Request. When we send a Request to an API, we have to specify what kind of action we want the API to perform with the data we are sending. In our [previous post](https://www.prostdev.com/post/understanding-apis-part-2-api-analogies-and-examples), we explained the calculator analogy and the Human Resources API example. Let me refresh your memory a bit. **Calculator analogy** In this analogy, we selected the calculator to act as our “API body” or API Implementation. We sent two numbers (2 and 2), and the Operation, in this case, was a sum. We got a result of 4 as the Output or Response. The calculator knows what to do with the numbers because we specify an Operation in the Request. ![Calculator analogy: 2 plus 2 request into a calculator returning 4 as the response](../../assets/blog/understanding-apis-part-3-what-are-http-methods-2.png) Breaking down the 4 aspects: ![Table breaking the calculator into Request and Response with Number data types and an addition operation](../../assets/blog/understanding-apis-part-3-what-are-http-methods-3.png) **Human Resources API** In this more practical example, we wanted to retrieve the first day of employment from certain employees. The list of employees was sent inside a CSV file containing the Employee ID, First Name, and Last Name. The Operation we specified for the API was a “GET.” This is what the Request looked like: ![Input CSV of two employees with the operation to GET their first day of employment](../../assets/blog/understanding-apis-part-3-what-are-http-methods-4.png) The Human Resources API then returned a new CSV file in which there was a new column called “First Day.” This contains the information we asked for in the Request, which was the employees’ first day of employment. ![Response CSV with an added First Day column highlighting each employee's start date](../../assets/blog/understanding-apis-part-3-what-are-http-methods-5.png) Breaking down the 4 aspects: ![Table of the HR API: CSV input and output with a GET operation](../../assets/blog/understanding-apis-part-3-what-are-http-methods-6.png) Now I’m sure you’re wondering why the Operation for this Human Resources API is just “GET.” Well, you’re about to figure it out! ## HTTP Methods As I mentioned earlier, a RESTful API communicates through HTTP. To communicate through HTTP, you need to specify what Operation you want the server to perform. When you write “google.com” in your browser and press Enter, you’re actually performing a GET method when trying to call Google - you want to get or retrieve the main Google website. The server then returns some data as the Response. Your browser is able to translate that information into a visual, which is the Google website with images and animations. And this is exactly what you were expecting. There are 9 HTTP methods: GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE, and PATCH. However, the most popular ones are GET, POST, PUT, PATCH, and DELETE. > [!NOTE] > There is A LOT of debate around the POST, PUT, and PATCH methods since they *can* be used to do “the same” kind of operations (create or update data). However, I will be based on the best practices shown by [Mozilla](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods) and [REST API Tutorial](https://www.restapitutorial.com/lessons/httpmethods.html). **GET** This method is used to read or retrieve data from the server or the API. We can see this in the Human Resources API example, when we wanted to retrieve the employees’ first day of employment. The information we sent in the Request was merely used to perform a search of the data, but no information was changed in the server. **POST** This method is used when you want to create new data. In the Human Resources API example, you would use POST to add a new employee. POST is **not** an [idempotent](https://en.wikipedia.org/wiki/Idempotence) method, which means that sending the same information more than once can result in duplicated data. This is why it’s used for creation only. **PUT** This method is mostly used to update or replace existing data, but it can also be used to create new data. The difference with POST is that PUT **is** an [idempotent](https://en.wikipedia.org/wiki/Idempotence) method, which means that sending the same information more than once will have the same effect as sending it just once. We’ll talk more about these differences in the following posts. **PATCH** This method is used to update or modify *parts* of the existing data. An example of this would be updating only the first name of an employee because there was a typo. PATCH *can* be an [idempotent](https://en.wikipedia.org/wiki/Idempotence) method, depending on how the API Implementation works. We’ll talk more about the differences between PUT and PATCH when we discuss updating data in the next posts. **DELETE** This is the easiest method to understand. This method is used to delete data, such as deleting an employee in the HR analogy. ## Recap - The answer to the quiz from the previous post (*What would be the Data Types for the RQ and RS in this scenario?)* is “Number.” - RESTful APIs or RESTful Web Services are calls that are made through HTTP (Hypertext Transfer Protocol). - There are 9 HTTP methods: GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE, and PATCH. However, the most popular ones are GET, POST, PUT, PATCH, and DELETE. - GET is used to retrieve data. - POST is used to create new data and is not an idempotent method (or unsafe). - PUT is mostly used to update or replace existing data but can also be used to create new data. It is an idempotent (safe) method. - PATCH is used to update or modify parts of the existing data and *can* be an idempotent method. - DELETE is used to delete data. As I’ve been saying throughout these posts, we’re absorbing information piece by piece. It’s ok if you’re still confused about what it all means or how it all fits together. We’ll get there; I promise. Just don’t give up! *Prost!* -Alex --- ## How to connect your Philips Hue Smart Lights with Google Calendar Source: https://prostdev.com/post/how-to-connect-your-philips-hue-smart-lights-with-google-calendar | Published: Oct 27, 2020 | Category: Tutorials If you’ve been working from home recently, you probably have had one of those dreaded WFH nightmares. You know the one I’m talking about. The one where you’re giving a presentation or you’re in the middle of a meeting with your webcam on, and your roommate or family suddenly enters the room. Wouldn’t it be great to be able to use your Philips Hue Smart Lights and synchronize them with your Google Calendar so that they show a red light when you’re busy? This way, they’ll know when you’re in the middle of a meeting and they won’t enter the room. For this post, I’ll guide you through some simple steps to synchronize your Philips Hue and Google account. ## Before starting If you're new to Philips Hue or you're thinking of purchasing, you should read this post first: [Things to consider before buying a Philips Hue Smart Light](https://www.prostdev.com/post/things-to-consider-before-buying-a-philips-hue-smart-light). I will be assuming that you already have your Google account (to use the Google Calendar) and your Smart Lights all set up with the [Hue application](https://www.philips-hue.com/en-ca/philips-hue-app). > [!NOTE] > This post will not guide you through the synchronization or installation of your Philips Hue Lights. You should also create a [Philips Hue account](https://account.meethue.com/) where you will be storing all the data from Hue - your lights, rooms, scenes, zones, etc. ## 1. Sign in to your Philips Hue account Go to [account.meethue.com](https://account.meethue.com/) and sign in with your credentials. If you go into the [Bridge tab](https://account.meethue.com/bridge), you should be able to see an online status for your bridge. ![Philips Hue account Bridge page showing the Hue Bridge status as online](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-1.png) You can also go into the [Apps tab](https://account.meethue.com/apps) to check some of the other applications that you may already have synced up with your Hue account. ## 2. Connecting IFTTT to your Google account Go into [ifttt.com](https://ifttt.com/home) (If This Then That) and create a new account using the Google account connected to your Google Calendar. You can simply click on [Sign Up](https://ifttt.com/join) and then choose the “Continue with Google” button. This will open a new browser window (or tab), so you can sign in to your Google account. This will automatically link your Google account to your brand-new IFTTT account. If you want to use a different email address from your Google account, you can create an IFTTT account with any email. Go into your [IFTTT account’s settings](https://ifttt.com/settings) and click on “Linked accounts” for Google. ![IFTTT Linked accounts settings showing Apple, Google, and Facebook all not linked](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-2.png) This will also open a new browser window (or tab), so you can sign in to your Google account. After doing this, your Google account will be linked. ![IFTTT Linked accounts now showing the Google account is linked with an Unlink option](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-3.png) ## 3. Configuring the trigger (If This) Click on the [Create button](https://ifttt.com/create/) on the top right side of the screen. This will open the screen where you will be able to program the integration (If This Then That). ![IFTTT "Create your own" screen with the "If This" Add button above "Then That"](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-4.png) Click on “If This” and search for Google Calendar. Click on the Google Calendar icon. ![IFTTT "Choose a service" search for "google calendar" showing the Google Calendar tile](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-5.png) Now select the “Any event starts” trigger. ![Google Calendar trigger options including "New event from search added" and "Any event starts"](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-6.png) Select the calendar you want to use from your Google account. You can also select how much time before the event starts that you want this integration to trigger. ![Trigger fields for "Any event starts" with a calendar picker and "Time before event starts" set to 0 minutes](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-7.png) Click on “Create trigger,” which will return you to the create screen so you can set up the “Then That” part. ## 4. Configuring the behavior (Then That) Click on this second button to continue with the configuration. ![IFTTT applet with the "If Any event starts" trigger set and "Then That" still to add](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-8.png) Now search for “Philips Hue” and click on the icon. ![IFTTT "Choose a service" search for "philips hue" showing the Philips Hue tile](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-9.png) You can choose any action you want your lights to perform. I’m going to choose “Change color.” ![Philips Hue action choices including Turn on/off lights, Blink, Dim, and Change color](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-10.png) Click on “Connect” to select your Philips Hue account. ![IFTTT "Connect service" screen for Philips Hue with a Connect button](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-11.png) This will open a new browser window (or tab), so you can sign in to your Philips Hue account. Select “Yes” to finish the setup. ![Philips Hue "Grant permission" page asking to trust IFTTT, with Yes and No buttons](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-12.png) Now you can select the light, room, or zone from your Philips Hue account that will change its color, and you can write the color name (ignore the “Add ingredient” button). Click on “Create action” when you’re done. ![Change color action fields with the Meeting Zone light selected and color set to Red](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-13.png) It’s all set up now! Just click on “Continue” to finish the configuration. ![Completed applet showing "If Any event starts, Then Change color" with a Continue button](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-14.png) You can also set a name so you can remember what this integration will do. You can choose whether you want to receive a notification when the Applet runs or leave it turned off. Click on “Finish.” ![Review and finish screen with the applet titled "Set Meeting Zone to Red when a new event starts"](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-15.png) ## 5. Did it work? Try it out! Create an event on your Google Calendar and wait for the lights to change. :) ![Google Calendar showing a test event from 8 to 9pm used to trigger the applet](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-16.png) Worked for me! ![A Philips Hue wall light glowing red after the calendar event triggered the applet](../../assets/blog/how-to-connect-your-philips-hue-smart-lights-with-google-calendar-17.png) You can check out all the different integrations or Applets that you can create in IFTTT - you’ll have a lot of fun! Let me know what else you create or if you have any questions. *Prost!* -Alex --- ## Things to consider before buying a Philips Hue Smart Light Source: https://prostdev.com/post/things-to-consider-before-buying-a-philips-hue-smart-light | Published: Oct 22, 2020 | Category: Opinion If you own an [Amazon Alexa](https://amzn.to/3JFoPwc) device or any of the [Google Assistant](https://assistant.google.com/) devices to create a smart home, you may already be looking into some smart lights to go with that! You may even be thinking about buying some [Philips Hue](https://www.philips-hue.com/) products such as smart bulbs, or maybe some lamps or smart switches. My first Philips Hue product was a [lightstrip](https://amzn.to/3mWJSk1)! I still have it, and I love it. But there are some things I wish I knew before buying it. In this post, I’ll show you some of the pros and cons of using Philips Hue for your smart home. ## What exactly are smart lights? There is one main difference between a regular light bulb and a smart bulb: you can turn on and off your regular bulb with only a wall switch. Smart bulbs can be turned on and off with a variety of smart switches, with a mobile app, with a voice command, or even with a synched schedule. Not just that, but you can also dim your smart bulbs and change their colors/tones. Sounding pretty cool so far, right? ## Choosing the right product - Bulbs You can choose to simply change your current regular bulbs with [Philips Hue Smart Bulbs](https://www.philips-hue.com/en-us/bulbs). You can choose the size you need, or if they’re indoor or outdoor bulbs too! When researching bulbs, something important to notice is that some are “white and color ambiance,” and some are just “white.” This small detail can be easy to miss at first, so make sure to choose the product you want. As you may have guessed, one bulb can change into different colors or tones, and the other one is just a white light (which can still be dimmed). ![Product cards comparing Hue White and color ambiance bulbs versus white-only Hue White bulbs](../../assets/blog/things-to-consider-before-buying-a-philips-hue-smart-light-1.png) Both are still smart lights, and both offer a range of functions. Just the colors of the light can’t be personalized for the Hue White bulbs. You can also choose some vintage bulbs, like this one: ![Hue White Filament vintage-style bulb product card with spec details](../../assets/blog/things-to-consider-before-buying-a-philips-hue-smart-light-2.png) Or some smaller “candle” bulbs, like these: ![Two small E12 candle bulb product cards, white ambiance and color ambiance versions](../../assets/blog/things-to-consider-before-buying-a-philips-hue-smart-light-3.png) These are just the most common bulbs to give you an example of the variety of products that you can find. You can go to the [product overview page](https://www.philips-hue.com/en-us/product-overview) and check out all the different lights they have to offer. The options are endless! ![Philips Hue product-overview grid of light types: bulbs, lightstrips, lamps, outdoor and more](../../assets/blog/things-to-consider-before-buying-a-philips-hue-smart-light-4.png) As you can see, you can buy just the bulbs, or you can also choose between different lamps or outdoor lightings. My favorite is, by far, the lightstrips. ## Lightstrips [Lightstrips](https://amzn.to/3mWJSk1) have some adhesive on the back, so you can paste them wherever you want! Check out these COOL examples from the internet: **Behind your desk** ![Source: Babbling Boolean YouTube channel - Philips Hue LightStrip Review](../../assets/blog/things-to-consider-before-buying-a-philips-hue-smart-light-5.jpg) **Over furniture / near the ceiling** ![Source: New Home Tricks - Philips Hue Lightstrip Plus](../../assets/blog/things-to-consider-before-buying-a-philips-hue-smart-light-6.jpg) **Light-up the halls!** ![Source: Trend Hunter - Customizable Connected Light Strips](../../assets/blog/things-to-consider-before-buying-a-philips-hue-smart-light-7.jpg) You can also use them behind the couch/furniture, behind a TV/screen, under the stairs, or under the table. You choose! ## Accessories So what are these [accessories](https://www.philips-hue.com/en-us/products#filters=CONTROLS_SU&page=1)? There are some smart switches that you can program from a mobile application to do different things - turn several lights on/off, dim the lights, choose different colors. You can do all these directly from the Hue app, but some people prefer to touch a switch. :) ![Hue accessory cards for the dimmer switch, smart button, and motion sensor](../../assets/blog/things-to-consider-before-buying-a-philips-hue-smart-light-8.png) There are also smart plugs, cable extensions, power supplies, and more. But wait! Don’t buy anything just yet, you still need to learn about some **very important** things. ## Connecting over Wi-Fi - Philips Hue Bridge All these lights can be used over Bluetooth, and for that, you only need your smartphone. But honestly, why use only Bluetooth when you can connect everything to the internet? If you connect your lights to your Wi-Fi, you can even turn on/off your lights when you’re away on vacation! But to do this, you need to make an additional purchase: the [Philips Hue Bridge](https://amzn.to/3FUbu0P). ![The white square Philips Hue Bridge hub with its status lights glowing blue](../../assets/blog/things-to-consider-before-buying-a-philips-hue-smart-light-9.png) > “The brains of the Philips Hue smart lighting system, the Hue Bridge allows you to connect and control up to 50 lights and accessories.” With this little box, you’ll be able to control up to 50 lights/bulbs/lamps and accessories. But you need to know this: **you need to connect your Bridge directly to your modem using an Ethernet cable**. You can’t just connect the Bridge to the Wi-Fi directly. Once the Bridge is connected with the Ethernet cable, *then* you can use all your products over Wi-Fi. *Sneaky marketing...* So, if your modem is not accessible to you for any reason, you may want to think twice before committing to Philips Hue. Here’s a quick video to get you started with your first Philips Hue Bridge installation and to get it connected to your modem: [Signify - How to install a Philips Hue starter kit with Jonathan Morrison](https://youtu.be/aTa26eNlI54). You can opt to buy the Bridge alone or choose one of the [Starter Kits](https://amzn.to/32Io5pF), which will provide you with the Bridge, plus some lights or switches. ## My favorite part - syncing your lights to your screen This is my absolute favorite thing about Philips Hue. You can sync the color of the lights in a given room based on whatever colors are on the screen. This gives you an immersive experience, especially when playing video games or watching movies! However… To be able to sync your lights to your screen, it is only free when using a PC/laptop through the [Hue Sync app](https://www.philips-hue.com/en-hk/entertainment/hue-sync), which is compatible with both a Mac OS and Windows. **If you want to do the same with a regular TV, you’ll need a** [Play HDMI Sync Box](https://amzn.to/3ztwLfw). I’ll end this post by providing this amazing video, so you can see this beautiful functionality in action (the more lights you use, the more detailed colors you’ll get): I hope this post was useful to help you choose your next smart lights. Let me know if you set them up! *Prost!* -Alex ## Links - [Amazon - Echo Dot](https://amzn.to/3JFoPwc) - [Google Assistant](https://assistant.google.com/) - [Philips Hue](https://www.philips-hue.com/) - [Amazon - Lightstrips](https://amzn.to/3mWJSk1) - [Amazon - Bulb](https://amzn.to/3pT9Irf) - [Philips Hue - Product Overview](https://www.philips-hue.com/en-us/product-overview) - [Amazon - Bridge (Hub)](https://amzn.to/3FUbu0P) - [Amazon - Starter Kit](https://amzn.to/32Io5pF) - [Philips Hue - Hue Sync app for PC/Laptop](https://www.philips-hue.com/en-hk/entertainment/hue-sync) - [Amazon - Play HDMI Sync Box](https://amzn.to/3ztwLfw) --- ## 3 Simple Steps to Convert a Flat File into JSON / CSV / XML Source: https://prostdev.com/post/3-simple-steps-to-convert-a-flat-file-into-json-csv-xml | Published: Oct 20, 2020 | Category: Tutorials Hello Muleys, This article is all about how simple it is to convert a Flat File into JSON, CSV, or XML using the all-time powerful weapon: DataWeave 2.0. Please do not forget to read the “Things to Remember” at the end of the article so that you don’t commit any mistake while coding. For this post, I used Anypoint Studio 7.4 and Mule Runtime version 4.2.2. Before starting, let's get familiar with a few terms. Flat File: (Also known as “FlatFile,” “FFD File,” or Flat-File”) A Flat File is data in a plain text format. Usually we get this kind of data from either mainframe systems, SFTP, or Files. Copybook: A copybook is a selection of code that defines data structures. If a particular data structure is used in many programs, then instead of writing the same data structure again, we can use copybooks. In general terms, to define the fields Data Types/Length and their structure, we define them in a copybook which in turn helps us in generating FFD files using a Transform Message. ## Step 1: Creating the copybook As previously said, a Flat File is a plain text string and the rows are separated by a new line, e.g. ``` 1234SRAVAN 27-09-1992TELANGANA HYDERABAD 11111 5678LINGAM 07-02-1996USA CHICAGO 77777 ``` This Flat File contains 98 characters with spaces (will discuss why spaces are also considered). Each row has 49 characters defined and they are divided by a new line. In my example, the data I am getting from SFTP or Mainframe basically contains the following information: ``` 'BIRTHID' 'BIRTHNAME' 'BIRTHDATE', 'BIRTHSTATE','BIRTHCITY', 'BIRTHZIP' ``` > [!IMPORTANT] > Remember the Flat File always has FIXED length! If you have BIRTHNAME defined for the length of 10 characters and if your name has only 6 characters, then the remaining 4 characters MUST be 4 empty spaces. Below is the copybook that I have created to define my incoming fields: ``` 000000 01 BIRTH-DETAILS. 000100 05 BIRTHID PIC X(4). 000200 05 BIRTHNAME PIC X(10). 000300 05 BIRTHDATE PIC X(10). 000400 05 BIRTHSTATE PIC X(10). 000500 05 BIRTHCITY PIC X(10). 000600 05 BIRTHZIP PIC X(5). ``` Please follow the indentation in this copybook. The type is nothing but your datatype of that particular field, length is the fixed length allocated to that field. Save this file with extension .cpy, e.g., BirthDetails.cpy. ## Step 2: Auto-generating the FFD file using DataWeave This is a pretty simple step with no coding involved! We basically need an FFD file to convert the Flat File to JSON or CSV. - Drag-and-Drop the Transform Message component in an empty flow. - Go to the Input section and click on Define Metadata. - Click on Add and give some Type Id. - Select Copybook from the Type drop-down. - Click Import and select the copybook that you have created in Step 1. - Finally, click Select. ![Anypoint Studio Select metadata type dialog importing BirthDetaiks.cpy as a Copybook type.](../../assets/blog/3-simple-steps-to-convert-a-flat-file-into-json-csv-xml-1.png) After clicking on Select, an FFD file will be created automatically and placed under `src/main/resources` > schemas > BirthDetails.ffd Please copy the FFD from the schemas folder and paste it inside `src/main/resources` as shown in below screenshot: ![Package Explorer with the generated BirthDetaiks.ffd file copied from schemas into src/main/resources.](../../assets/blog/3-simple-steps-to-convert-a-flat-file-into-json-csv-xml-2.png) FFD file generated: ```yaml form: COPYBOOK id: 'BIRTH-DETAILS' values: - { name: 'BIRTHID', type: String, length: 4 } - { name: 'BIRTHNAME', type: String, length: 10 } - { name: 'BIRTHDATE', type: String, length: 10 } - { name: 'BIRTHSTATE', type: String, length: 10 } - { name: 'BIRTHCITY', type: String, length: 10 } - { name: 'BIRTHZIP', type: String, length: 5 } ``` ## Step 3: Final Transformation No matter what your data source is; whether it’s a File Read connector or an HTTP Listener, the first thing that you need to place after the source is a Set Payload component. Drag-and-Drop a Set Payload component and configure as below. It needs to have a value of #[payload] Now go to the MIME Type tab and configure recordTerminator and schemaPath as below: ![Set Payload MIME Type tab set to application/flatfile with recordTerminator and schemaPath parameters.](../../assets/blog/3-simple-steps-to-convert-a-flat-file-into-json-csv-xml-3.png) Place a Transform Message just after the Set Payload and code it as below: ![Transform Message mapping each copybook field to a JSON details object with default "" values.](../../assets/blog/3-simple-steps-to-convert-a-flat-file-into-json-csv-xml-4.png) If you want your output in a CSV format, just replace application/json with application/csv If you want your output in XML format, we need another Transform Message after application/json output because this doesn't have a root tag. If you try to map directly to application/xml, it will throw an error! That's it finished :) ## Things to Remember From the following image you can see that the name is defined as 10 characters in your FFD file. But "sravan' has only 6 characters. So the input name must have 10 characters, i.e., ``` "sravan " ``` We have to have 4 spaces appended at the end. This is because for each row, we defined the FFD file with the following rules: - First 4 characters belong to Birth ID - Next 10 characters belong to Birth Name - Next 10 characters belong to Birth Date - And so on... Each special character is treated as 1 character. The rows must be divided using a new line break (highlighted in yellow). If you can see the input, the State name for the second row is USA (3 characters). So we have appended 7 spaces to it to make it a perfect and valid Flat File. One of the validations mentioned above failed in input, we will get Transformation errors! ![Flat file input with padding spaces highlighted, transformed into two JSON records on the right.](../../assets/blog/3-simple-steps-to-convert-a-flat-file-into-json-csv-xml-5.png) Try it out! Pass the Flat File data through Postman using a POST Method and select the type as Text. Send it as input for your Application. ![Postman POST request with the flat file pasted in a raw Text body, ready to send to the app.](../../assets/blog/3-simple-steps-to-convert-a-flat-file-into-json-csv-xml-6.png) Here’s your final Mule application: ![Final Mule flow: HTTP Listener, then Set Payload, then Transform Message.](../../assets/blog/3-simple-steps-to-convert-a-flat-file-into-json-csv-xml-7.png) Get back to me if you have any doubts! Happy Learning! Yours, Sravan Lingam --- ## 7 tips to start writing your technical blog post Source: https://prostdev.com/post/7-tips-to-start-writing-your-technical-blog-post | Published: Oct 13, 2020 | Category: Opinion In the previous post, [Things to consider before writing a technical blog post](https://www.prostdev.com/post/things-to-consider-before-writing-a-technical-blog-post), we asked ourselves some questions to help us choose the topic that we want to write about. We also learned how to visualize our target audience before writing our post. For this post, we will start writing our article or blog post! Yes, just like that. Take your laptop, your PC and your keyboard, your tablet, or whatever you’re most comfortable using. Just take it and follow these simple steps to start writing! ## 1. Pick a title You can change this title later, but pick something that’ll keep you focused on what you’re trying to explain. This way, if you find yourself over-explaining something, just read your title once more and ask yourself if you actually need that long paragraph, or if you can keep it shorter since it’s not the focus of your article right now. You can create a title that explains exactly what you will be talking about. Look at this example: “How to get started on Java programming.” You can also create lists of something related to the technology you chose, for example: - “5 ways to improve your application’s performance” - “10 tips for improving your time management skills” - “5 reasons why you should be using Service Mesh” - “The 10 biggest mistakes when developing Python scripts” - “The 5 biggest misconceptions of RESTful APIs” Some authors choose titles that are catchy or end with a question to grab the reader’s attention, like this one: [Software Quality is not cheap, but would you risk your company’s future for the lack of it?](https://www.prostdev.com/post/software-quality-is-not-cheap-but-would-you-risk-your-company-s-future-for-the-lack-of-it) Ultimately, you want a title that will help your audience understand the topic of the post, but you also want something memorable. ## 2. Write an introduction Most people will only read the first(s) paragraph(s) before they decide if your post is worth reading or not. The introduction can be just one or two paragraphs; try to keep it simple. Write a summary of what you’re going to cover with your article and try to get the reader’s attention. A popular way of ending an introduction is with one or two questions that you’re going to answer throughout the post, which will also pique the reader’s curiosity. - “Can this function be used with objects?” - “How can you achieve high performance with complex code?” - “Is it really worth using APIs for your project?” ## 3. Write some definitions or pre-requisites This step is optional. It depends on what kind of audience is reading your post. I personally try to explain the basic concepts that the reader needs to know so that they can follow along with me. How long the explanation is, depends on the definition’s complexity and how important it is for the audience to know this term before they read the rest of the post. Sometimes I use two or three sentences to define a term or concept. Sometimes I use one or two paragraphs. And sometimes, I even create a whole separate article to explain it, and then I just reference that post (with a link) before starting my current one. ## 4. Develop the article Now you can start writing the actual article’s content. Explain all you want, or talk to the reader just as if you were face to face. Use formal or informal wording. In this part, try to be yourself! This is what will differentiate your post from others. Even if you write about a very well-known subject, maybe you’re using a new perspective that no one has done before, or maybe you’re giving easy-to-read explanations. This is your essence as a writer. Own it! Make use of titles and subtitles, or bold and underlined formatting, in order to have a readable and user-friendly format. Structure your post in sections. If your audience has to pause reading the article for a meeting, when they return, they’ll easily want to know where to continue reading afterwards. Having subtitles or sections will help them maintain some order. Otherwise, they’ll get lost in between paragraphs and may lose interest in your post. > [!NOTE] > people tend to read articles that are shorter (2-5 minutes, or 800-1,300 words), but sometimes posts can become longer when there are a lot of explanations involved. You can always separate your information in a series of posts (Part 1, Part 2…) instead of creating a huge article. ## 5. Add visual content Most people nowadays want to read articles because they are way quicker than reading a book. We don’t have the time, nor the mind, to be able to sit for hours and read a programming language book. Instead, we read articles or watch videos that can easily break down the information. For this same reason, the average reader today may lose interest if the post is only paragraphs with no visual content. Adding media (pictures, gifs, pieces of code) will definitely get the reader’s attention, and people may focus on these visuals if they don’t want to read everything. Or the readers can skip the explanations because they can understand your post just by looking at the pictures or even the subtitles. Be careful not to add a ton of media everywhere; doing this will also lose the reader’s attention because they can lose the sense of the section they’re in and feel lost. You do *not* want your target audience to feel lost while reading your article. ## 6. Write a conclusion or a summary Some readers will skip your article altogether because they don’t feel like reading. It’s not your fault; that's just the way it is. So, be ready for your readers to skip the whole explanation. Create one or two paragraphs of a *very quick* summary of the complete article - maybe one or two sentences per definition. If they don’t understand the summary completely, they’ll know that it’s because they skipped the detailed explanation. Another way to look at it is: Imagine that your reader is about to get to an important meeting and they need to sound like they know everything about your article, so you have 30 seconds to explain it to them; otherwise, they’re going to get fired! ## 7. Say goodbye This, just like the body of the post, has to sound like you. This is where you get to be friendly and yourself. You can use one or two sentences to say goodbye and thank them for reading. You can even sign with your name or a nickname. I personally prefer to sign with “-Alex.” ## Recap - **Pick a title** that will catch the reader’s attention. You can choose to explain exactly what the article is about, make a list of things, or create a catchy title. You can even add a question to the title to spark your audience’s curiosity. - **Write an introduction** to give the reader a better idea of the post’s topic. This is where they’ll decide if they want to read your content or not! - **Write some definitions or pre-requisites**, so your reader is not confused when they read the rest of the post. Depending on who the article is for (your target audience), you can give a quick 2-sentence summary, or you can redirect the reader to a previous article. - **Develop the article**! Remember to be yourself and try to keep it short, unless it’s a very technical post. You can also break it down into a series of articles instead of having one long post. - **Add visual content** to keep your reader’s attention, but don’t overdo it! - **Write a conclusion or a summary**. Remember to imagine that you have 30 seconds to summarize your article, or your colleague will get fired! - **Say goodbye** to the reader in your own special way. I hope this was helpful to get you started on creating some blog posts! Leave me a comment if you have any questions or if you’d like to see more content about writing technical articles. *Prost!* -Alex --- ## Implementing NetSuite Saved Search with MuleSoft Source: https://prostdev.com/post/implementing-netsuite-saved-search-with-mulesoft | Published: Oct 6, 2020 | Category: Tutorials ## Introduction NetSuite is a SaaS-based ERP which allows companies to manage important business using a single tool. NetSuite provides a suite of cloud-based financials / Enterprise Resource Planning (ERP) and omnichannel commerce software. NetSuite has capabilities of ERP, CRM, Manufacturing, e-commerce, retail. MuleSoft provides a NetSuite connector, which enables to automate the business process and synchronize the data between NetSuite and third-party tools. NetSuite Connector uses SuiteTalk WSDL to provide SOAP-based integration to generate NetSuite business objects, make use of different authentication levels, and support error handling. NetSuite connector provides various features and capabilities: - SOAP-Based integration. - Connecting NetSuite using REST calls to RESTlets that expose APIs created with SuiteScript. - Multiple types of Authentication. - Error Handling support. ## How to Connect NetSuite with MuleSoft MuleSoft NetSuite connector provides various ways of Authentication to connect NetSuite. - Login authentication. - Request based authentication. - SSO authentication. - Token authentication. ![NetSuite connector config dropdown listing Login, Request, SSO, and Token authentication](../../assets/blog/implementing-netsuite-saved-search-with-mulesoft-1.png) One of the most common ways of connecting NetSuite is using Token authentication. For this, we will need Consumer key, Consumer secret, Token ID, and Token secret. ![NetSuite token authentication form with consumer key, secret, token, and account fields](../../assets/blog/implementing-netsuite-saved-search-with-mulesoft-2.png) Now, we are going to see a few operations related to NetSuite. ## Saved Search Saved Search allows us to search the data in NetSuite and it can be performed on any record type in NetSuite. MuleSoft provides the operation Search which can be used to perform the search on the record type. ## Use Case 1 - Basic Employee Search (lastName starts with J) Let's consider, we want to retrieve all the Employee details from NetSuite whose lastName starts with "J". MuleSoft’s search operation in the NetSuite connector can be used to retrieve all employee details whose lastName starts with "J". We will be adding the following script in MuleSoft Transform Message to pass as Payload to NetSuite connector for the search operation and record type as `EMPLOYEE_BASIC`. ```dataweave %dw 2.0 output application/java --- { lastName: { operator: "STARTS_WITH", searchValue: "J" } } as Object { class: "org.mule.module.netsuite.extension.api.EmployeeSearchBasic" } ``` **NetSuite Connector Configuration** ![NetSuite Search operation config with key EMPLOYEE_BASIC and search record set to payload](../../assets/blog/implementing-netsuite-saved-search-with-mulesoft-3.png) ## Use Case 2 - Advanced Employee Search Let's consider, we want to retrieve all the employee details from NetSuite. NetSuite has created Saved Search which can be utilized in the MuleSoft application to fetch all employee details. NetSuite provided us savedSearchId as "8656" and savedSearchScriptId as "EmployeeSearch8656". We will be adding the following script in MuleSoft Transform Message to pass as Payload to NetSuite connector for the search operation and record type as `EMPLOYEE_ADVANCED`. ```dataweave %dw 2.0 output application/java --- { savedSearchId: "8656", savedSearchScriptId: "EmployeeSearch8656" } as Object { class: "org.mule.module.netsuite.extension.api.EmployeeSearch" } as Object { class: "org.mule.module.netsuite.extension.api.EmployeeSearchAdvanced" } ``` **NetSuite Connector Configuration** ![NetSuite Search operation config with key EMPLOYEE_ADVANCED and search record set to payload](../../assets/blog/implementing-netsuite-saved-search-with-mulesoft-4.png) ## Use Case 3 - Advanced Employee Search (lastModifiedDate is after 1st August 2020) Let's consider, we want to retrieve all the Employees details from NetSuite whose lastModifiedDate is after 1st August 2020. NetSuite has created Saved Search which can be utilized in the MuleSoft application to fetch all employee details whose lastModifiedDate is after 1st August 2020. NetSuite provided us savedSearchId as "8657" and savedSearchScriptId as "EmployeeSearch8657". We will be adding the following script in MuleSoft Transform Message to pass as Payload to NetSuite connector for the search operation and record type as `EMPLOYEE_ADVANCED`. ```dataweave %dw 2.0 output application/java --- { savedSearchId: "EmployeeSearch8657", savedSearchScriptId: "EmployeeSearch8657", criteria: { basic: { lastModifiedDate: { operator: "AFTER", searchValue: "2020-08-01T12:00:00" as LocalDateTime {format: "yyyy-MM-dd'T'HH:mm:ss"} } as Object { class: "org.mule.module.netsuite.extension.api.SearchDateField" } } as Object { class: "org.mule.module.netsuite.extension.api.EmployeeSearchBasic" } } as Object { class: "org.mule.module.netsuite.extension.api.EmployeeSearch" } } as Object { class: "org.mule.module.netsuite.extension.api.EmployeeSearchAdvanced" } ``` ## Use Case 4 - Advanced Employee Search (isInactive is false and retrieve firstName, middleName and lastName) Let's consider, we want to retrieve all the Employees details from NetSuite who are active and retrieve only firstName, middleName, and lastName. We will be adding the following script in MuleSoft Transform Message to pass as Payload to NetSuite connector for the search operation and record type as `EMPLOYEE_ADVANCED`. ```dataweave %dw 2.0 output application/java --- { columns: { basic: { firstName: [{ customLabel: "First Name" }], middleName: [{ customLabel: "Middle Name" }], lastName: [{ customLabel: "Last Name" }] } }, criteria: { basic: { isInactive: { searchValue: false } } } } as Object { class: "org.mule.module.netsuite.extension.api.CustomerSearchAdvanced" } ``` ## Use Case 5 - Basic employee search (lastModifiedDate is after 1st August 2020 and isInactive is false) Let's consider, we want to retrieve all the Employees details from the NetSuite who are active and lastModifiedDate is after 1st August 2020. We will be adding the following script in MuleSoft Transform Message to pass as Payload to NetSuite connector for the search operation and record type as `EMPLOYEE_BASIC`. ```dataweave %dw 2.0 output application/java --- { lastModifiedDate: { operator: "AFTER", searchValue: "2020-08-01T12:00:00" as LocalDateTime {format: "yyyy-MM-dd'T'HH:mm:ss"} }, isInactive{ searchValue: false }, } as Object { class: "org.mule.module.netsuite.extension.api.EmployeeSearchBasic" } ``` Now, you know how to perform search operations with the MuleSoft NetSuite connector. --- ## What is CIDR (Classless Inter-Domain Routing) in MuleSoft VPC Source: https://prostdev.com/post/what-is-cidr-classless-inter-domain-routing-in-mulesoft-vpc | Published: Sep 29, 2020 | Category: Guides ## Introduction CIDR stands for Classless Inter-Domain Routing and it’s a way of allocating IP addresses or hosts in a more efficient manner. It replaces the old way of allocating IP addresses based on the class system. This method allocates the IP Addresses or hosts in a more efficient way and avoids the waste of IP Addresses. - Class A: 16 million host identifiers or IP Addresses. - Class B: 65,535 host identifiers or IP Addresses. - Class C: 254 host identifiers or IP Addresses. Let's consider the Organization requires around 500 IP Addresses or Hosts. In such cases, organizations have to go with a Class B IP distribution system where almost more than 60,000 IP addresses are wasted. ## What is an IP Address? IP Addresses consist of two groups of bits in the address, the most significant bits are network the prefix, which identifies the network or subnet; and the least significant bits are the host identifier, which specifies a particular interface of the host on that network. IP Addresses have 2 components: - Network Address - Host Address Each IP Address (IPv4) is 32 bit or 4 Octet. Below is the representation of an IP Address in Binary: ![Binary breakdown of the IP address 192.168.0.1 across its four 8-bit octets.](../../assets/blog/what-is-cidr-classless-inter-domain-routing-in-mulesoft-vpc-1.png) CIDR Block Notation: ``` xxx.xxx.xxx.xxx/n ``` where n is the number of bits used for the subnet mask. The subnet mask is made up of setting up all network bits to all ones and host bits to all zeros. Let's consider this scenario: if you provide this CIDR Block 192.168.0.0/24, it will give 255 hosts or IP addresses. ![Table mapping CIDR notation to total hosts: /24 gives 256, /23 gives 512, /22 gives 1024.](../../assets/blog/what-is-cidr-classless-inter-domain-routing-in-mulesoft-vpc-2.png) ## What is a Subnet? A subnetwork or subnet is a logical subdivision of an IP network. The practice of dividing a network into two or more networks is called subnetting. Computers that belong to a subnet are addressed with an identical most-significant bit-group in their IP addresses. Now, we will see how to Calculate the total number of hosts using Subnet Mask. ## Use Case 1 Subnet Mask 192.168.0.0/24 will equate to IP Range 192.168.0.0 — 192.168.0.255. ![Logical AND of IP 192.168.0.1 with the /24 subnet mask, showing 24 network bits and 8 host bits.](../../assets/blog/what-is-cidr-classless-inter-domain-routing-in-mulesoft-vpc-3.png) N represents Network and H represents Host. In the above example, we made 24 bits to ones and the remaining 8 bits to zeros because the Subnet Mask end range is 24. Total zeros are 8 for Host (2*2*2*2*2*2*2*2=256). This will give an IP range of 192.168.0.0 — 192.168.0.255 (256 Hosts in total). ## Use Case 2 Subnet Mask 192.168.0.0/23 will equate to IP Range 192.168.0.0 — 192.168.0.511. ![Logical AND of IP 192.168.0.1 with the /23 subnet mask, showing 23 network bits and 9 host bits.](../../assets/blog/what-is-cidr-classless-inter-domain-routing-in-mulesoft-vpc-4.png) N represents Network and H represents Host. In the above example, we made 23 bits to ones and the remaining 9 bits to zeros because the Subnet Mask end range is 23. Total zeros are 9 for Host (2*2*2*2*2*2*2*2*2=512). This will give an IP range of 192.168.0.0 — 192.168.0.511 (512 Hosts in total). ## Use Case 3 Subnet Mask 192.168.0.0/27 will equate to IP Range 192.168.0.0 — 192.168.0.31. ![Logical AND of IP 192.168.0.1 with the /27 subnet mask, showing 27 network bits and 5 host bits.](../../assets/blog/what-is-cidr-classless-inter-domain-routing-in-mulesoft-vpc-5.png) N represents Network and H represents Host. In the above example, we made 27 bits to ones and the remaining 5 bits to zeros because the Subnet Mask end range is 27. Total zeros are 5 for Host (2*2*2*2*2=32). We have borrowed 3 bits from the host to make a total of 27 bits. Subnet will be (2*2*2=8) and the Host will be 32. So we can get a total of 8 subnets. Subnetworks will be 192.168.0.0/27, 192.168.0.31/27, 192.168.0.63/27, 192.168.0.95/27, 192.168.0.127/27, 192.168.0.159/27, 192.168.0.191/27, 192.168.0.223/27 Here we are dividing the subnet mask into smaller subnetworks. Whenever you are creating MuleSoft VPC, you need to make sure whatever CIDR Mask you are providing doesn't conflict with your on-premise or any other networks. The smallest network subnet block you can assign for your Anypoint VPC is /24 and the largest is /16. For each worker deployed to CloudHub, the following IP assignation takes place: - For better fault tolerance, the VPC subnet may be divided into up to four Availability Zones. - A few IP addresses are reserved for infrastructure. - At least two IP addresses per worker to perform at zero-downtime. ## MuleSoft VPC Sizing Now, we learn how we can do the VPC sizing. Below are some requirements. - You have four environments: dev, test, sit and prod. - Application on dev and sit must run on 1 Worker. - Application on test must run on 2 Workers. - Application on prod must run on 2 Workers. - Total Application = 100 (Near Future). - The organization will have 2 VPCs: one for PROD and another for NON-PROD. The problem is that we need to decide the minimum CIDR block that will be needed for PROD and NON-PROD VPCs. ![VPC sizing table per environment totaling 300 IPs for Production and 600 for Non-Production.](../../assets/blog/what-is-cidr-classless-inter-domain-routing-in-mulesoft-vpc-6.png) There will be 2 IPs reserved for each VPC for infrastructure. For Production VPC, we require around 300 IPs and it will be provided by a subnet mask of /23 (e.g. 192.168.0.0/23). This subnet mask will provide 512 IPs. For Non-Production VPC, we require around 600 IPs and it will be provided by a subnet mask of /22 (e.g. 192.168.0.0/22). This subnet mask will provide 1024 IPs. You should know how you can make use of the CIDR range efficiently and perform MuleSoft VPC sizing. --- ## Mule 4 Continuous Integration using Azure DevOps Source: https://prostdev.com/post/mule-4-continuous-integration-using-azure-devops | Published: Sep 24, 2020 | Category: Tutorials *GitHub repository with the project can be found at the end of the post.* Once we start developing applications in MuleSoft and we keep our code stored in any source control platform like Github, Bitbucket, GitLab, or Azure; as a developer and as a best practice, we look into automating the process to deploy our applications to either CloudHub or an On-Premise server. In this post I will explain how a MuleSoft application can be automatically deployed into CloudHub or an On-Premise server from Azure DevOps as our main Continuous Integration (CI) and source control platform. ## Creating a new project in Azure The first step would be to setup our project in Azure DevOps, for this you need a Microsoft account. You can get one [here](https://dev.azure.com/). Then, we can create a new project, provide a name and a description, as well as set the privacy settings. By default you can set it as "private". ![Azure DevOps Create a project form with name mulesoft-pipeline and Private visibility selected](../../assets/blog/mule-4-continuous-integration-using-azure-devops-1.png) Once it is created, we can go into the main page and locate the "Repos" section. Once we see the page, we can use the information to send our code directly from our local environment. ![Empty Azure Repos page showing clone URL and commands to push an existing repository](../../assets/blog/mule-4-continuous-integration-using-azure-devops-2.png) Just before actually pushing any code we can create a pretty simple endpoint that we can hit later to verify it’s all working. Let's create a simple RAML definition in Anypoint Platform: ```yaml #%RAML 1.0 title: Sample API /persons: get: responses: 200: body: application/json: example: | [{ "id": 1, "name": "Edgar Moran", "username": "emoran", "email": "yucel.moran@some.test", "address": { "street": "Kulas Light", "suite": "Apt. 556", "city": "Gwenborough", "zipcode": "92998-3874", "geo": { "lat": "-37.3159", "lng": "81.1496" } }}] ``` Then we can generate the flows in our project. Right click on the RAML file > Mule > Generate Flows from Local REST API. ![Anypoint Studio right-click menu choosing Mule then Generate Flows from Local REST API](../../assets/blog/mule-4-continuous-integration-using-azure-devops-3.png) Once we have our flow generated, we can proceed to create our `test.yaml` file in order to identify any value we want to attach to our environment. It will look just like this: ```yaml http: port: "8081" environment: "[TEST]" ``` I will create a `global.xml` configuration file to keep my configurations. I will add a Configuration Properties element: ![Configuration properties dialog with the File field set to an env-based YAML placeholder](../../assets/blog/mule-4-continuous-integration-using-azure-devops-4.png) After that, we can create a Global Property, in this case called env which will allow us to know which environment we are working on. ![Global Property dialog defining a property named env with a placeholder value](../../assets/blog/mule-4-continuous-integration-using-azure-devops-5.png) I know, a lot of steps, right? But let's try to make it right, just before our final check to verify everything is working. Let's add an environment variable called env in our Run Configuration. ![Anypoint Studio Run As submenu with Run Configurations option highlighted](../../assets/blog/mule-4-continuous-integration-using-azure-devops-6.png) ![Run Configurations Environment tab adding an env variable set to test](../../assets/blog/mule-4-continuous-integration-using-azure-devops-7.png) Now let's run the project. If everything was correctly set up, you'll see your project deployed! ![Anypoint Studio flow with APIkit Router and console showing the app deployed successfully](../../assets/blog/mule-4-continuous-integration-using-azure-devops-8.png) We can start pushing our first version of this working code into our Azure repository, so let's do that. ## Generating a personal access token to push our code to Azure Let's go back into the Azure DevOps main project page: ![Azure DevOps mulesoft-pipeline project overview page with summary and stats](../../assets/blog/mule-4-continuous-integration-using-azure-devops-9.png) And let's create a personal access token: ![Azure DevOps user menu open with Personal access tokens option highlighted](../../assets/blog/mule-4-continuous-integration-using-azure-devops-10.png) Click con Create a new token. You'll see a bunch of options, select the actions and permissions you consider are the best depending on your needs. ![Create a new personal access token panel with name, expiration, and scope checkboxes](../../assets/blog/mule-4-continuous-integration-using-azure-devops-11.png) Once created, you'll have a new token which becomes your project password, so keep it safe! Now we can push our code. If the terminal asks you for a password, use the token you just generated and your code should be on master. ![Azure Repos Files view on master listing src, .gitignore, mule-artifact.json and pom.xml](../../assets/blog/mule-4-continuous-integration-using-azure-devops-12.png) All right, so we have our code already in Azure. Now let's create a new branch for our repository, just let's call it test based from master and let's create another one as a developer branch called emoran. It's important to mention that during this sample on emoran branch we will review the build and in the test branch we will deploy to CloudHub. ## Creating an Artifact Before creating our pipeline we need to make sure we have the right information in our `settings.xml` and our `pom.xml` files. This artifact will allow us to get the Distribution Management piece we need to set in our project. For this, we will need to go into our main project page, and locate the artifact option on the left, then we need to click on "Connect to feed". We select maven and then it will show the information we need to put in place in our `settings.xml` and `pom.xml` files. ![Azure Artifacts Connect to feed page showing Maven setup XML for pom and settings files](../../assets/blog/mule-4-continuous-integration-using-azure-devops-13.png) ## Preparing the settings.xml and pom.xml files The next task will be to prepare our files in order to make sure we will be able to build and deploy our application into CloudHub. Here's how my `settings.xml` file looks like: ```xml yucelmoran-azure-pipeline yucelmorans ${azure.password} MuleRepository ${nexus.username} ${nexus.password} Mule true yucelmoran-azure-pipeline https://pkgs.dev.azure.com/yucelmorans/yucelmoran-azure-pipeline/_packaging/yucelmoran-azure-pipeline/maven/v1 true true MuleRepository MuleRepository https://repository.mulesoft.org/nexus-ee/content/repositories/releases-ee/ default true true ``` and here's the `pom.xml`: ```xml 4.0.0 com.emoran yucelmoran-azure-pipeline 1.0.0-SNAPSHOT mule-application yucelmoran-azure-pipeline UTF-8 UTF-8 4.3.0 3.3.5 1.7 scm:git:git@ssh.dev.azure.com:v3/yucelmorans/yucelmoran-azure-pipeline/yucelmoran-azure-pipeline scm:git@ssh.dev.azure.com:v3/yucelmorans/yucelmoran-azure-pipeline/yucelmoran-azure-pipeline https://yucelmorans@dev.azure.com/yucelmorans/yucelmoran-azure-pipeline/_git/yucelmoran-azure-pipeline yucelmoran-azure-pipeline https://pkgs.dev.azure.com/yucelmorans/yucelmoran-azure-pipeline/_packaging/yucelmoran-azure-pipeline/maven/v1 org.apache.maven.plugins maven-clean-plugin 3.0.0 org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true https://anypoint.mulesoft.com 4.3.0 ${anypoint.username} ${anypoint.password} Sandbox yucelmoran-azure-pipeline Micro true ${env} deploy deploy deploy org.codehaus.mojo buildnumber-maven-plugin 1.1 buildnumber validate create {0,number} buildNumber false false unknownbuild org.mule.tools.maven mule-app-maven-plugin ${mule.tools.version} true org.apache.maven.plugins maven-install-plugin 2.5.2 org.apache.maven.plugins maven-deploy-plugin 2.8.2 org.mule.connectors mule-http-connector 1.5.17 mule-plugin org.mule.connectors mule-sockets-connector 1.1.6 mule-plugin org.mule.modules mule-soapkit-module 1.2.6 mule-plugin org.mule.modules mule-apikit-module 1.3.13 mule-plugin yucelmoran-azure-pipeline https://pkgs.dev.azure.com/yucelmorans/yucelmoran-azure-pipeline/_packaging/yucelmoran-azure-pipeline/maven/v1 true true MuleRepository MuleRepository https://repository.mulesoft.org/nexus-ee/content/repositories/releases-ee/ default true true mulesoft-release mulesoft release repository default http://repository.mulesoft.org/releases/ false mulesoft-release mulesoft release repository default http://repository.mulesoft.org/releases/ false MuleRepository MuleRepository https://repository.mulesoft.org/nexus-ee/content/repositories/releases-ee/ default ``` Basically, we are specifying where is our repository for source control, and then we specify where we want to deploy our application with the cloudHubDeployment configuration into the mule-maven-pluging tag. We can verify everything works using this command: ```bash mvn clean install -s settings.xml -e ``` ## Creating a pipeline Now we can create our pipeline configuration. Find the Pipelines options and select "Create Pipeline." ![Azure Pipelines section with the Create your first Pipeline prompt and button](../../assets/blog/mule-4-continuous-integration-using-azure-devops-14.png) Select the Azure Repos Git option and select your repository from your project. ![New pipeline Where is your code step with the Azure Repos Git YAML option](../../assets/blog/mule-4-continuous-integration-using-azure-devops-15.png) In the next step we can select Maven and then a template yaml file would be generated (we will change this eventually). ```yaml # Maven # Build your Java project and run tests with Apache Maven. # Add steps that analyze code, save build artifacts, deploy, and more: # https://docs.microsoft.com/azure/devops/pipelines/languages/java trigger: - master pool: vmImage: 'ubuntu-latest' steps: - task: Maven@3 inputs: mavenPomFile: 'pom.xml' mavenOptions: '-Xmx3072m' javaHomeOption: 'JDKVersion' jdkVersionOption: '1.8' jdkArchitectureOption: 'x64' publishJUnitResults: true testResultsFiles: '**/surefire-reports/TEST-*.xml' goals: 'package' ``` When you finish, the new yaml file will appear as part of your code. We need to modify the script, leaving it this way: ```yaml # Maven # Aurthor: Edgar Moran # Build your Java project and run tests with Apache Maven. # Add steps that analyze code, save build artifacts, deploy, and more: # https://docs.microsoft.com/azure/devops/pipelines/languages/java trigger: - test - emoran pool: vmImage: 'ubuntu-latest' steps: - task: Maven@3 condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/test')) inputs: mavenPomFile: '$(system.defaultWorkingDirectory)/pom.xml' mavenOptions: '-Xmx3072m' javaHomeOption: 'JDKVersion' jdkVersionOption: '1.8' jdkArchitectureOption: 'x64' publishJUnitResults: true testResultsFiles: '**/surefire-reports/TEST-*.xml' goals: 'clean deploy -Danypoint.username=$(ANYPOINT_USERNAME) -Danypoint.password=$(ANYPOINT_PASSWORD) -Denv=test -Dazure.password=$(AZURE_TOKEN) -Dnexus.username=$(NEXUS_UERNAME) -Dnexus.password=$(NEXUS_PASSWORD) -s settings.xml -e' - task: Maven@3 inputs: mavenPomFile: '$(system.defaultWorkingDirectory)/pom.xml' mavenOptions: '-Xmx3072m' javaHomeOption: 'JDKVersion' jdkVersionOption: '1.8' jdkArchitectureOption: 'x64' publishJUnitResults: true testResultsFiles: '**/surefire-reports/TEST-*.xml' goals: 'clean install -Dazure.password=$(AZURE_TOKEN) -Dnexus.username=$(NEXUS_UERNAME) -Dnexus.password=$(NEXUS_PASSWORD) -s settings.xml -e' ``` It’s important to set the variables we will use during deployment, for example: credentials to login to nexus repositories in case we need to, credentials for Anypoint Platform and the personal token we created in Azure. Once we create our pipeline we can select the option Variables. ![Pipeline YAML editor showing the generated azure-pipelines.yml with Maven trigger config](../../assets/blog/mule-4-continuous-integration-using-azure-devops-16.png) Finally, let's create as many as we need, according to what we need from our yaml file. ![Pipeline Variables panel listing Anypoint, Azure token, and Nexus credential secrets](../../assets/blog/mule-4-continuous-integration-using-azure-devops-17.png) Now the last piece of the puzzle is basically to send a change from source control to Azure on the first step (sending a change to the emoran branch, for example). ![Terminal git status on the emoran branch showing the modified azure-pipeline.xml file](../../assets/blog/mule-4-continuous-integration-using-azure-devops-18.png) ![Terminal output of git push origin emoran creating the new branch on Azure](../../assets/blog/mule-4-continuous-integration-using-azure-devops-19.png) We'll see our pipeline running for emoran. ![Azure pipeline run summary for the emoran branch with a job still running](../../assets/blog/mule-4-continuous-integration-using-azure-devops-20.png) When it finishes, we'll see a *clean install build* command for this branch. ![Pipeline Maven job log for the emoran branch finishing a clean install build](../../assets/blog/mule-4-continuous-integration-using-azure-devops-21.png) Now all happens in source control, so if we want to deploy our application to CloudHub, then, we need to promote our changes from emoran to the test branch. Let's create a Pull Request in Azure, from emoran to test: ![Azure DevOps New pull request form merging the emoran branch into test](../../assets/blog/mule-4-continuous-integration-using-azure-devops-22.png) Once we click on Create, then we need to complete the Pull Request and Merge Request in order to trigger the pipeline. When the pipeline that we ran for test finishes, we can see the deployment happened this time: ![Pipeline Maven job log for the test branch uploading the artifact and deploying](../../assets/blog/mule-4-continuous-integration-using-azure-devops-23.png) So now we can go to Anypoint Platform and see our app deploying. ![Anypoint Runtime Manager listing the yucelmoran-azure-pipeline app with Updating status](../../assets/blog/mule-4-continuous-integration-using-azure-devops-24.png) ![Runtime Manager detail panel showing the deployed CloudHub app in Started status](../../assets/blog/mule-4-continuous-integration-using-azure-devops-25.png) Finally, with this you can set as many branches you need and try other processes along. ## GitHub repository Hope this helps others to set their process. You can find the complete code [here](https://github.com/emoran/mule4-yucelmoran-azure-pipeline). Hope to hear from you on comments and let me know if this was helpful. --- ## Implementing Mapping Rules With MuleSoft Dedicated Load Balancer Source: https://prostdev.com/post/implementing-mapping-rules-with-mulesoft-dedicated-load-balancer | Published: Sep 22, 2020 | Category: Guides ## Introduction Dedicated Load Balancer is an optional component within the Anypoint Platform and it is used to route the HTTP and HTTPS traffic to multiple applications deployed to CloudHub workers in the VPC. To create a Dedicated Load Balancer, you must first create the Anypoint VPC which can be mapped to multiple environments and the same Dedicated Load Balancer can be used for different environments. You can use multiple DNSs for the same Dedicated Load Balancer (i.e. `api-dev.example.com` and `api-test.example.com`) ## Applying Mapping Rules on a Dedicated Load Balancer (DLB) Mapping rules are used on dedicated load balancers to translate input URI to call applications deployed on CloudHub. A pattern is a string that defines a template for matching an input text. Whatever value is placed within curly brackets ({ }) is treated as a variable. Variable names can contain only lowercase letters (a-z) and no other characters, including slashes. Let's consider that we have 2 DNS (i.e. `api-dev.example.com` and `api-test.example.com`) setup on a dedicated load balancer: `api-dev.example.com` is for the Dev environment whereas `api-test.example.com` is for the Test environment. ## Use Case 1 We are receiving requests on the DLB `https://api-dev.example.com/ecommerce/v1.0/invoice` and need to redirect them to `http://org-ecommerce-api.cloudhub.io/v1.0/invoice` (the CloudHub application name will be `org-ecommerce-api`) We can use this mapping rule to achieve this: ![DLB mapping rule table with an app path variable mapped to an org-app-api target, output /v1.0, protocol http](../../assets/blog/implementing-mapping-rules-with-mulesoft-dedicated-load-balancer-1.png) This previous rule will be applied when requests come on DLB and route to the CloudHub application in the VPC. ``` https://api-dev.example.com/ecommerce/v1.0/invoice (DLB) ``` is redirected to: ``` http://org-ecommerce-api.cloudhub.io/v1.0/invoice (CloudHub) ``` ![Use Case 1](../../assets/blog/implementing-mapping-rules-with-mulesoft-dedicated-load-balancer-2.png) But here we have some problems that on our DLB, we have set up 2 DNSs, one for Dev and another for Test. Now, how will the DLB know this is a request that needs to route to either the Dev or Test application because the same rule will be applied for both? To avoid this, we will be using a subdomain in the next use case. ## Use Case 2 In this case, we will be using a subdomain for routing the request to the correct environment from DLB. Our application name format must be org-app-subdomain (e.g. org-ecommerce-api-dev for dev environment and org-ecommerce-api-test for test environment) when deploying to CloudHub workers in VPC. So, our mapping rule will look like this: ![DLB mapping rule using app and subdomain variables in the target, output /v1.0](../../assets/blog/implementing-mapping-rules-with-mulesoft-dedicated-load-balancer-3.png) subdomain is a variable to map any subdomain. **1) dev environment** ``` https://api-dev.example.com/ecommerce/v1.0/invoice (DLB) ``` is redirected to: ``` http://org-ecommerce-api-dev.cloudhub.io/v1.0/invoice (CloudHub Dev Environment) ``` **2) test environment** ``` https://api-test.example.com/ecommerce/v1.0/invoice (DLB) ``` is redirected to: ``` http://org-ecommerce-api-test.cloudhub.io/v1.0/invoice (CloudHub Test Environment) ``` ![Use Case 2](../../assets/blog/implementing-mapping-rules-with-mulesoft-dedicated-load-balancer-4.png) In this use case, we solve the issue of routing the request from DLB to the correct environment. Let's consider another scenario where you want to route the request to CloudHub on the basis of the application version. We will see this in the next use case. ## Use Case 3 In this case, we will deploy an application to CloudHub and it will be in format org-app-subdomain-version (e.g. org-ecommerce-api-dev-v1-0 for Dev environment and org-ecommerce-api-test-v1-0 for Test environment). Whenever we get a request on the DLB, then the version in the URL will be v1.0 and v2.0 but when you deploy an application on CloudHub it doesn't allow you to use "." in the application name. That is the reason we are using "-" in the version of the application deploying to CloudHub. So, our mapping rule will look like this: ![DLB mapping rule with subdomain and version variables in the input and target paths](../../assets/blog/implementing-mapping-rules-with-mulesoft-dedicated-load-balancer-5.png) **1) dev environment** ``` https://api-dev.example.com/ecommerce/v1.0/invoice (DLB) ``` is redirected to: ``` http://org-ecommerce-api-dev-v1-0.cloudhub.io/v1.0/invoice (CloudHub Dev Environment) ``` **2) test environment** ``` https://api-test.example.com/ecommerce/v1.0/invoice (DLB) ``` is redirected to: ``` http://org-ecommerce-api-test-v1-0.cloudhub.io/v1.0/invoice (CloudHub Test Environment) ``` ![Use Case 3](../../assets/blog/implementing-mapping-rules-with-mulesoft-dedicated-load-balancer-6.png) ## DLB Mapping Rules Priority DLB will apply the first matching rule regardless of more exact matching rules available. A rule defined first, at index 0 has higher priority against other rules defined after it. The higher the index assigned, the less priority the mapping rule has. This is how you can apply mapping rules on MuleSoft's Dedicated Load Balancer. --- ## Securing APIs: Two ways to apply policies for CloudHub applications Source: https://prostdev.com/post/securing-apis-two-ways-to-apply-policies-for-cloudhub-applications | Published: Sep 15, 2020 | Category: Tutorials > [!NOTE] > Anypoint Platform’s API Manager policies can only be applied to the applications which are deployed to CloudHub. For other deployment modes you need to have other ways of applying policies. This article helps you to apply policies via Anypoint Platform’s API Manager. We have 2 ways to apply policies: - Creating Proxy APIs - Without Creating Proxy APIs The main difference between the two of them is: - Using API ID in Auto-Discovery of your “actual” application, - Or Mule itself will create a Proxy application on top of your “actual” application which has API Auto-Discovery. ## 1) Without Creating Proxy API We will need these two things: - API ID generated in API Manager and used in the global element “Auto-Discovery” of your application. - Anypoint Platform’s Client ID and Client Secret. We can get them from “Access Management” Under Environments Tab > Choose the environment in which your application is getting deployed. Use them in Runtime properties while deploying the application in below way: ``` anypoint.platform.client_id : xxxxxxxxxxxx anypoint.platform.client_secret : xxxxxxxxxxxx ``` With the help of these two steps, we can now see in API Manager that the status of our API will turn from “Unregistered” to “Active”. **Step 1**: Design a RAML, publish the asset to Exchange, import the RAML to Anypoint Studio, and generate the flows. **Step 2**: Go to API Manager, Click on Manage API and choose “Manage API from Exchange”. ![API Manager "Manage API" dropdown with the "Manage API from Exchange" option highlighted](../../assets/blog/securing-apis-two-ways-to-apply-policies-for-cloudhub-applications-1.png) **Step 3**: Select your API, and select the managing type as “Basic Endpoint”. Don’t forget to check the box “Mule version: Check this box if you are managing this API in Mule 4 or above.” Click on Save. ![Manage API from Exchange form for BankDetails with managing type set to Basic Endpoint](../../assets/blog/securing-apis-two-ways-to-apply-policies-for-cloudhub-applications-2.png) ![BankDetails API instance page with status Unregistered and the API Instance ID highlighted](../../assets/blog/securing-apis-two-ways-to-apply-policies-for-cloudhub-applications-3.png) **Step 4**: Go to the Application in Anypoint Studio, Create a Global Element Auto-Discovery and Add API id and choose the flow name (usually Main flow where the request listens). ![Anypoint Studio API Autodiscovery global element with the API Id and Flow Name filled in](../../assets/blog/securing-apis-two-ways-to-apply-policies-for-cloudhub-applications-4.png) **Step 5**: Go to Access Management (the owner of the account can have access and can view this option. If you are not seeing this, that means you need to request access to the owner of the Anypoint Platform account). Choose the Environments Tab and select your environment. Get your Client ID and Client Secret. ![Anypoint Platform navigation menu with the Access Management option highlighted](../../assets/blog/securing-apis-two-ways-to-apply-policies-for-cloudhub-applications-5.png) ![Access Management Environments with the Sandbox "Edit environment" panel showing Client ID and Secret](../../assets/blog/securing-apis-two-ways-to-apply-policies-for-cloudhub-applications-6.png) **Step 6**: Go to Runtime Manager and add the Runtime properties like below and deploy. ![Runtime Manager Properties tab with anypoint.client_secret and anypoint.client_id set as text properties](../../assets/blog/securing-apis-two-ways-to-apply-policies-for-cloudhub-applications-7.png) Now your API in API manager status will turn to Active and you can start applying policies by going to Policies > Apply New Policy. > [!NOTE] > There’s no need to restart the application once the policy is applied. It will reflect within seconds automatically. Even if you add one or more policies at a later point of time, no restart of the application is required. ## 2) Creating Proxy API In the previous method there were a lot of manual processes required. To avoid all these, we can go for this second approach to create a Proxy application. This Proxy application is nothing but a kind of wrapper to your main application. Now for each application, we will have 2 apps in Runtime Manager: - Your Actual Application - Your Proxy Application, which has API ID, Client ID, and Client Secret within it. There’s no manual effort required to deploy. It’s deployed from the API manager. **Step 1**: Design a RAML, publish the asset to Exchange, import the RAML to Anypoint Studio, and generate the flows. **Step 2**: Go to API Manager, Click on Manage API and choose “Manage API from Exchange”. **Step 3**: Select your API, and select the managing type as “Endpoint with Proxy”. Don’t forget to check the box “Mule version: Check this box if you are managing this API in Mule 4 or above.” Click on Save. **Step 4**: Give the Implementation URL of your actual application that is deployed. Make sure you provide your complete path, up until /api (all RAML-generated flows have a common base path as /api) E.g. ``` http://myfirstapp.cloudhub.io/api ``` Once done, click on Save. ![Proxy setup form with deployment target CloudHub selected and the Implementation URI field filled in](../../assets/blog/securing-apis-two-ways-to-apply-policies-for-cloudhub-applications-8.png) **Step 5**: Now you can see the Deployment Configuration. Choose the runtime and name of the Proxy app you want to give and click on deploy. The application will be deployed in the Runtime Manager. ![API Manager Deployment Configuration with runtime version 4.2.0 and a proxy application name set](../../assets/blog/securing-apis-two-ways-to-apply-policies-for-cloudhub-applications-9.png) **Step 6**: Now edit the Consumer endpoint in API Manager with your Proxy endpoint. So that all the requests will hit the proxy application. Basically your implementation URL is your actual endpoint and Consumer endpoint is the endpoint for the Proxy. In other words, the client hits the Proxy endpoint, which in-turn is routed to the implementation URL. ![Active API instance showing the implementation URL, the consumer endpoint, and the deployed proxy URL](../../assets/blog/securing-apis-two-ways-to-apply-policies-for-cloudhub-applications-10.png) You can see that the Client ID and Client Secret values are picked automatically and placed in Runtime properties. You can also download the jar of the Proxy application and import it in Anypoint Studio, where you can see the API ID is automatically configured and the HTTP request will route to your main app. You have to provide the endpoint of the Proxy application to consumers/clients so that they don’t have access to your main app. The policies are applied to your Proxy application. But the thing here is, we need additional vCore and workers to deploy our Proxy app. According to your requirements, you can choose which method you want to follow to apply policies for your CloudHub apps. ## Pros and Cons Without Proxy Application: - You will save a vCore and worker. - As a developer, you need to know the Client ID and Client Secret - unless the deployment is managed by CI/CD process. - You need to manually create a global element to invoke Auto-Discovery. - You will expose your actual endpoint to the Client/Consumer. - Use case: If your APIs are used within the organization, within intranet, you can choose this option where we can share the endpoints directly. This saves vCores. With Proxy Application: - You will need additional vCore and worker for each Proxy app you deploy. - As a developer, you don’t need to know the Client ID since it’s managed internally. - No need for the manual process of using API ID or using Client ID and Client Secret. - You will expose your Proxy endpoint over the actual endpoint to the Client/Consumer. - Use case: If your APIs are used outside of your organization, you can choose this option where we can share the endpoints of the Proxy app instead of the actual endpoint. This gives you more security. These are 2 simple ways to apply policies! Happy learning Sravan Lingam --- ## Things to consider before writing a technical blog post Source: https://prostdev.com/post/things-to-consider-before-writing-a-technical-blog-post | Published: Sep 8, 2020 | Category: Guides Congratulations on taking the first step towards writing! You may have chosen to do this because you’re curious about trying something new, or maybe you feel relaxed when writing and you want to take up a new hobby. Maybe you noticed that it could be a good thing on your resume, or maybe it’s just a great opportunity for you to acquire new knowledge and skills for your professional career. Whatever your reason for writing may be, I just want to let you know that you’re not alone. A lot of us have felt like writing, but were not sure where to start or what to do exactly. For this post, I will give you some tips on how you can get started and get your first article out! ## Choosing what to write about > [!NOTE] > When I say “technology,” it can mean a programming language, a software development tool, a mobile application, a computer program, etc. Basically, it is anything related to Software. (At least in ProstDev) you don’t need to be an absolute expert on a technical topic to be able to write about it. It would help if you had more information about it, in case you need to explain certain details for the article, but you can also write about your own personal experience with a technology. Think about any technology that makes you happy, or that you really enjoy using. Maybe quite the opposite, think about a technology that was hard to use and understand and create a blog post about everything you learned through your experience to make someone else’s life easier. You can start by asking yourself some questions to get ideas about a technology or an experience that you’d like to share with a wider audience: - Was there a technology in your professional career that you always wanted to learn more about, but you never did? - Was there a technology that was exceptionally fun to learn? - Was there a technology that you struggled *a* *lot* to learn, but in the end, it proved to be super useful? - Was there a technology that you truly didn’t want to learn, but then you kept seeing it as a requirement for other job positions? - Was there a technology that you were really terrible at, but you still thought it was interesting and wanted to learn more about it? - Is there a technology that you thought wasn’t going to be that important when you first heard about it, but now it’s imperative to know it for your current role? - Is there a technology that you want to keep using for your software development projects because it’s very useful? - Is there a technology that you thought was *nearly* perfect, but you stopped using it because of those small flaws? - Is there a technology that you thought would be revolutionary for the industry, but then everyone stopped using it, and it became “unimportant”? - Is there a technology that you keep suggesting to use in your project, but your company or the client won’t let you make this change? - Do you excel in a specific type of technology? - Was there a technology in your university/college days that you absolutely enjoyed and were good at, but then you never got to use it in the professional world? - Was there a technology you learned during your internship(s) or your first job(s)? - Is there a specific topic from any technology that you really struggled to understand and had to do a lot of research to master it? - Is there a specific problem you encountered using any technology, and you couldn’t find any documentation online to solve it, but in the end, you were able to troubleshoot it on your own? I hope that you have a list of technologies or features in your mind now! If not, just keep asking yourself more questions and you’ll get to a technology or a topic that would make a great blog post. If that doesn’t work, you can also just start by writing about the technology you’re currently using, whether for software development, architecture, design, management, or even some methodologies that you use for your team like Scrum, Kanban, or Waterfall. Choose the first topic you want to write about, and let’s go to the next step! ## Who to write to (a.k.a. your target audience) Now you need to start thinking about who will be reading your article. It’s good to create a blog post just for the sake of writing, but it’s way better if it’s actually useful for other people. One way to create better content is to define the target audience; this way, you will have a specific group of people interested in reading your post. Ask yourself, who will read this article? Try to be as specific as you can, so you can also write your content accordingly. Here are some more questions you can ask yourself to define your audience: - Is this for developers? What kind of developers? (Front-end, back-end, integration, data scientists, etc.) - Is this for architects? - Is this for business people? Are they project managers? - Are these people new to the technology? Are they just starting to learn it? - Do these people hold senior positions, or have a lot of years of experience in their position? You’re not going to write complex code or technical details if your target audience is someone from the business side. You’re not going to give an introduction to a programming language if you’re writing for senior developers - they will be expecting lines of code to see the solution! One trick that I do to stay focused on writing for my target audience is to actually imagine someone that I know from that area and write as if I was face to face with them explaining my article. This helps me add some definitions that they may not be familiar with, so they can follow the rest of the post (for basic posts). Or this keeps me focused on *not* explaining the basics and just get straight to the point (for advanced topics). Now we know what to write about and who to write the blog post for. In my next post, I will give you more tips to actually start writing your article. So, stay tuned for things like picking a title, organizing the post’s format, or adding additional content. Subscribe to ProstDev to receive an email as soon as this new content is published! *Prost!* -Alex --- ## Understanding APIs (Part 2): API Analogies and Examples Source: https://prostdev.com/post/understanding-apis-part-2-api-analogies-and-examples | Published: Sep 3, 2020 | Category: Guides In Part 1 of this series, “[Understanding APIs (Part 1): What is an API?](https://www.prostdev.com/post/understanding-apis-part-1-what-is-an-api),” we learned about the acronym, the definition, the four aspects of an API, and how everything works together. We also defined this cute little diagram below. If you feel like you forgot some pieces, you can always go back and read the recap at the end for a quick summary. ![Diagram: a Request input flows through an Implementation box to a Response output](../../assets/blog/understanding-apis-part-2-api-analogies-and-examples-1.png) When I started learning what an API was, it took me some time to really start understanding this concept. It can become so abstract! When I look at practical examples of applied technologies, it can really open my eyes to understanding a theory. In the case of APIs, looking at different examples helped me get a better idea of this concept. This post will complement what you learned in Part 1. You will learn about some analogies and real-life examples of APIs to understand better what they are. ### API Analogies Before I begin, I just want to get something clear: when you are using or *consuming* an API, you don’t really care about how the Implementation works. It doesn’t matter if the Implementation is built on Java, Python, or MuleSoft; it’s still going to give you the Response that you are expecting according to your Request. You don’t care about what systems the API needs to call, if any, or if the API is using JSON or CSV as the internal Data Type. None of this is really visible to you anyways. You just want your Request to be processed and your Response to be returned. **1. Restaurant** Imagine you arrive at a restaurant. You sit in a chair and start looking at the menu. Then the server comes to ask for your order. You say that you want a hamburger because it was one of the choices that you selected from the menu. The employee goes to the kitchen, and someone cooks your meal. Sometime later, the server comes back with your hamburger. *“How is this an analogy for an API?”* You, the client, give a Request to the server: you want to receive a hamburger. The server does whatever needs to be done to deliver the order. Then, you receive a Response with what you asked for: a hamburger. ![Restaurant analogy: a client sends a menu request and a server returns a dish response](../../assets/blog/understanding-apis-part-2-api-analogies-and-examples-2.png) Notice how I didn’t add the part of the server going to the kitchen, or the chef cooking your meal. This is because we only care about what our order was and what was delivered to us. **2. Calculator** Now, let’s say that a regular calculator is an API. You send a “2+2” Operation as a Request, hit the equal button, and receive a “4” as a Response. ![Calculator analogy: a 2+2 request goes into a calculator and returns 4 as the response](../../assets/blog/understanding-apis-part-2-api-analogies-and-examples-3.png) Once more, you’re not interested in how the calculator processed this information; you just want an accurate response to the Inputs and Operation that you sent in the Request. **Quiz**: What would be the Data Types in the RQ and RS in this scenario? *The answer will be revealed in the next post.* ### API Examples Now let’s see some examples of how APIs can be used in real-life scenarios. **1. Human Resources API** You started by being a customer at a restaurant, then a calculator user, and now you are the head of the Human Resources (HR) department at FictionalCorp. Hooray! As the HR head manager, you obviously know how to use the HR API to process the company’s information regarding the employees. You want to check what the employees’ first day of employment at FictionalCorp was, and you already know that the HR API accepts Requests in the format of a CSV file. Your Request would look like this: ![CSV request with two employees and an operation to get their first day of employment](../../assets/blog/understanding-apis-part-2-api-analogies-and-examples-4.png) There’s a CSV Data Type, the Input information of the two employees (*we’re a startup, ok?*), and the Operation telling the API what to do with this data. You click “Send,” and off it goes! Within seconds the API returns the Response with the information you requested: ![CSV response adding a highlighted First Day column with each employee's start date](../../assets/blog/understanding-apis-part-2-api-analogies-and-examples-5.png) It returns the same Data Type (CSV format) and an Output file that contains the same information you sent in the Request, PLUS, a new field called “First Day,” which contains each of the employees’ first day of employment at FictionalCorp. You may not know if the API is connected to a Database or to an external system where this information is kept, but it returned what you requested in the first place, which is all you need. What would happen if the RQ/RS was in a different format, you say? Well, this HR API also accepts the [JSON](https://www.json.org/json-en.html) Data Type. Here’s how that would look: ![Same HR request and response shown as JSON, the response adding a FirstDay field](../../assets/blog/understanding-apis-part-2-api-analogies-and-examples-6.png) We have the same Inputs, same Operation, and same Output, but the Data Types for both the Request and the Response are different. **2. Real-life APIs** Do you want to see some actual APIs? Follow these links to see their documentation: - [Twitter API](https://developer.twitter.com/) - [Slack API](https://api.slack.com/) - [Google Maps API](https://developers.google.com/maps/documentation) - [Facebook APIs](https://developers.facebook.com/) It’s ok if you don’t understand everything that you see in these pages. This is just to give you an idea of what real APIs look like. ## Recap To recap, when you’re calling or *consuming* an API, you don’t really care about how the Implementation works; you just want your Request to be processed and a Response to be returned according to the Inputs and Operations that you sent in the RQ. Stay tuned for Part 3 of this series if you want to know the answer to the quiz in the second analogy: the calculator. The question was: *What would be the Data Types for the RQ and RS in this scenario?* I hope you are starting to get a better picture of what APIs are. Contact me or leave me some comments if you have any additional questions - I’m happy to help. Keep learning and don’t give up! *Prost!* -Alex --- ## Creating a connector and consuming it from a Mule Application Source: https://prostdev.com/post/creating-a-connector-and-consuming-it-from-a-mule-application | Published: Sep 1, 2020 | Category: Tutorials ## What is a Connector? An Anypoint Connector is a reusable component that interacts with Mule runtime and Anypoint Studio. A connector communicates with a target resource and conveys information between a resource and Mule, and transforms the data into a Mule message. [Anypoint Connector DevKit](https://docs.mulesoft.com/connector-devkit/3.9/) abstracts the communication that happens between Mule and the external service or API, and generates a user interface to help simplify usage of the connector by the developer who would eventually use it in their application. A well-developed connector makes the development of a Mule application much easier for users when handling tasks like pagination, session expirations, and input and output metadata. This tutorial shows you how to create a well-designed connector. ## Why create a Custom Connector? In our MuleSoft journey we, as developers, may have used different connectors when creating Mule applications. We have a good number of connectors available out-of-the-box for us to use. There could be a scenario where no out-of-the-box connector is available and we need to follow a series of complex steps (i.e. authorizing the client, getting access token, implement caching, calling database, etc.) to actually call a third party API. In this scenario, it is recommended to use a custom connector which will be responsible for calling the third party API and hiding the complexity altogether. Consumer API would otherwise have no idea how that particular third party API is called by the connector. ## Creating a Custom Connector In Mule 3, we had a separate API Dev Kit to create an Anypoint Connector Project directly from Anypoint Studio. However, with Mule 4, that is no longer required. Creating a connector project has become simpler than ever. **Prerequisites**: - Java JDK 1.8 - Maven Apache Plugin Run the following command with maven. This should create a basic template for the connector project with test cases. This will create a project with sample operations and basic structure with required files. However, you can change the implementation anytime according to your needs. ```bash mvn org.mule.extensions:mule-extensions-archetype-maven-plugin:1.2.0:generate ``` This will ask for some basic parameters like name and version of the project. If you don’t want to specify anything just press Enter or the Return key (for Mac users) leaving it empty. Maven will take the default values for that. Some important parameters are: - **Name of the extension**: specify the name you want to set for the connector. - **GroupId**: If you are planning to deploy it to Exchange, you need to provide Organization Id as groupId. If your extension is certified by Mule, then you can use `org.mule.modules` which is the default value in case you do not provide anything. - **Version**: you can provide a version number here. If you leave it empty, it will take 1.0.0 - SNAPSHOT as the default value. - **ArtifactId**: Provide a name for the artifact. You can keep the same as the extension name (the first point). - **Package**: You can specify the Java package namespace here. If you leave it empty, it will take the default one: `org.mule.extensions.project.name.internal` for Java implementation and `org.mule.extensions.project.name.test` for testing. ![Maven archetype command output prompting for connector name, groupId, and version, ending in BUILD SUCCESS](../../assets/blog/creating-a-connector-and-consuming-it-from-a-mule-application-1.png) ## How to Obtain Organization Id To get the Organization Id for your Anypoint Platform instance; log-in to your Anypoint Platform Account, navigate to Access Management >> Organization, and click on the name of the business group. A new pop-up window will open where you can find the Organization Id. ![Anypoint Access Management Organization info dialog with the Organization Id highlighted](../../assets/blog/creating-a-connector-and-consuming-it-from-a-mule-application-2.png) ## Configure the POM File for the Connector This step is crucial. We will make some changes in the connector’s POM file to make it ready for publishing to Anypoint Exchange. First add a mule-maven-plugin like below: ```xml org.mule.tools.maven mule-maven-plugin ${mule.maven.plugin.version} true mule-application-example ``` Then create a profile with the exchange distribution repository information. ```xml exchange ExchangeRepository Corporate Repository https://maven.anypoint.mulesoft.com/api/v1/organizations/${groupId}/maven default ``` Please note, `groupId` is a variable which is extracted from the groupId properties from this section of the POM file. ```xml 4.0.0 c5f216cc-80a0-45ee-9721-8a0809356e37 mule-test-connector 1.0.5 mule-extension mule-test-connector 3.3.5 ``` Finally, you need to add the server section in the `settings.xml` file from your `./m2` folder. You can use your username and password. Please note that the password is encrypted, because I am using `settings-security.xml` that contains an encrypted master password which is used for decryption of server password. The details for encrypting your password can be found in this amazing blog post: [How to Encrypt your Maven Password](https://dev.to/scottshipp/how-to-encrypt-your-maven-password-408d). However, for testing, you can stick with a plain password if you want to. ```xml ExchangeRepository YourUserName {PCdHLVg1NrFdHY+ighZBQF4/DlJYCjxYxA=} ``` Please note that `server.id` should be the same as the `repository.id` mentioned in the profile section above. ## Java Project Structure Once you create your Mule connector project which is basically a Java project and can be opened in any Java [IDE](https://www.prostdev.com/post/5-reasons-why-you-need-an-ide) (i.e. IntelliJ Idea). Once you open the project in a Java IDE, you should see the project structure as shown in the picture. ![IDE project tree for mule-test-connector showing the Configuration, Extension, and Operations classes](../../assets/blog/creating-a-connector-and-consuming-it-from-a-mule-application-3.png) Please note that `org.mule.extension.mule.test.internal` is your Java package namespace that was mentioned during the creation of the connector project. Any connector project has the 3 following class files: - **<project-name>Configuration**: It contains the configuration parameters. Whatever parameters we mention here will be used to create the connector’s configuration in Mule. - **<project-name>Extension**: This is the main extension file and starting point of the connector. - **<project-name>Operations**: This file contains the methods for the connector’s operations to do the specific task. This contains the actual connector operation’s implementation. Each connector operation is actually a method in this class file. ![Java code for the connector's HelloWorld and Calculate operations annotated with @MediaType and @DisplayName](../../assets/blog/creating-a-connector-and-consuming-it-from-a-mule-application-4.png) If you want unit tests designed for a connector, this project will contain some additional files for that purpose. ## Publish and Consume the Connector from MuleSoft Once we have everything in place in our Java Connector project, we need to deploy this to Anypoint Exchange. ```bash mvn deploy -Pexchange -DskipTests ``` Here with the -P parameter, we need to specify the profile name as mentioned in the POM file. ![Command prompt running mvn deploy -Pexchange to publish the connector](../../assets/blog/creating-a-connector-and-consuming-it-from-a-mule-application-5.png) This should deploy the connector in Anypoint Exchange. You can see the asset in Anypoint Exchange as shown in the screenshot. ![Anypoint Exchange Assets view showing the published Mule-test-Extension connector by Jyoti Choudhury](../../assets/blog/creating-a-connector-and-consuming-it-from-a-mule-application-6.png) Now it’s time to consume that connector from the Mule project. We simply start by adding dependencies in the POM file. ```xml c5f216cc-80a0-45ee-9721-8a0809356e37 mule-test-connector 1.0.5 mule-plugin ``` Please change the groupId, artifactId, and version based on your connector project POM file. If everything is in place, maven dependencies will be resolved and the connector will be added as a component. Finally, we can use this custom connector like any other out-of-the-box connector that’s available in the market. ![Studio flow with Listener, the custom Hello world connector operation, and Transform Message](../../assets/blog/creating-a-connector-and-consuming-it-from-a-mule-application-7.png) ## Conclusion We have learnt so far from this post: - What is Connector and why do we use Custom Connector? - Learn to create Custom Connector project. - Learn to publish and consume Custom Connector from MuleSoft API. Hope this has helped you. If you have any questions, please feel free to post your questions. ## References - [https://docs.mulesoft.com/connector-devkit/3.9/devkit-tutorial](https://docs.mulesoft.com/connector-devkit/3.9/devkit-tutorial) - [https://dzone.com/articles/how-to-publish-mule-custom-connector-to-anypoint-e](https://dzone.com/articles/how-to-publish-mule-custom-connector-to-anypoint-e) - [https://javastreets.com/blog/publish-connectors-to-anypoint-exchange.html](https://javastreets.com/blog/publish-connectors-to-anypoint-exchange.html) Thanks. Jyoti --- ## Understanding APIs (Part 1): What is an API? Source: https://prostdev.com/post/understanding-apis-part-1-what-is-an-api | Published: Aug 27, 2020 | Category: Guides I know that the concept of an “API” can be a bit hard to understand. It’s such an abstract thing! If you’re in technology, you probably have heard already about these APIs, but you may not fully understand what it means or what it is just yet. Hey, don’t worry. The same thing happened to me! It took me some time to really figure out what these are. For this post, I’ll give you a simple explanation and a cool diagram for you to refer to whenever you get confused with the terminology. ## What *is* an API? First of all, API is an acronym for Application Programming Interface. You can read the complete wiki definition [here](https://en.wikipedia.org/wiki/Application_programming_interface). Now that the textbook definition is out of the way, let me start by stating that an API can be created in different programming languages or different technology tools. Let’s call the “body” of the API, the *Implementation*. ![Examples of programming languages and tools that can be used to create an API.](../../assets/blog/understanding-apis-part-1-what-is-an-api-1.png) An API is composed of 4 specific aspects (plus the Implementation of the *actual* API). - Inputs - Operations - Outputs - Data Types **1. Inputs** The Inputs for the API are pieces of data or information that you can send to the Implementation. **2. Operations** This is where you define what you expect the API to do with the Inputs you will be sending to the Implementation. **3. Outputs** These are also pieces of data or information, but instead of *sending* these to the API (or Implementation), you will be *receiving* these *from* the API (or Implementation). **4. Data Types** The data or information that you send to the API and that you receive from it must be of some type. It can be numbers, words, an Excel file, a Java Object, a CSV file, a JSON or XML format, etc. You can send, for example, some input numbers and tell the API which operation you expect it to process, and get some more numbers in return as the output. Maybe you send your name as an input, and you receive a “Hello” plus your name as the output. All these formats are the Data Types. ## Defining an API's Request and Response **Request** The combination of Inputs (along with their Data Types) and Operations is what we call a “Request.” In other words, everything that you send to the API or the Implementation is a Request (RQ). **Response** The Outputs from the API or Implementation, as well as their Data Types, are what we call a “Response.” Put simply, everything sent back from the API to you, the caller, is a Response (RS). ## Putting it all together ![Diagram: Request (Input with Data Type + Operation) flows into Implementation, returning a Response (Output with Data Type).](../../assets/blog/understanding-apis-part-1-what-is-an-api-2.png) The Request is formed by the Inputs we want to send to the API. This includes their specified Data Types and the specific Operation that we want the API to process and return to us. This Request is sent to the Implementation of the API, which is processed by the code, and then the Outputs with the specified Data Types are returned to us. ## Um… Ok… Not quite sure what an API is yet? APIs are a very abstract concept, as I mentioned before, and I wouldn’t blame you if you went through this complete post and still felt confused as to what an API is. Not to worry, my friend! I thought of that too. I wanted to have this explanation clarified first, so you could at least keep in mind this last diagram. Save this picture somewhere because you may need to re-reference it as you keep learning. As you may have guessed from the title, this is only Part 1 of this series! We will continue to go on a journey through the wonders and philosophies of APIs. [Part 2](https://www.prostdev.com/post/understanding-apis-part-2-api-analogies-and-examples) will give you some analogies and examples of APIs. This way, you’ll end up with a better picture of this weird, but needed, concept. Let me just finish up with a quick recap. You are allowed to forget what you learned; it happens to everyone! The important thing is to keep trying and don’t give up. ## Recap - API is an acronym for Application Programming Interface. - The “body” of the API is also called the *Implementation*. - APIs are composed of Inputs, Operations, Outputs, and Data Types. - The Request is what we *send* to the API: Inputs with their Data Types and Operations. - The Response is what the API *returns* to us: Outputs with their Data types. That’s it for now! You can subscribe to this blog to receive an email as soon as new content is published. You can also follow ProstDev on social networks to see more of our content. :) *Prost!* -Alex --- ## Reviewing Sorting Algorithms: Merge Sort Source: https://prostdev.com/post/reviewing-sorting-algorithms-merge-sort | Published: Aug 25, 2020 | Category: Guides Let’s sort it out! In a series of posts, we will be discussing some of the sorting algorithms listed in the below order: - [Bubble Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-bubble-sort) - [Selection Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-selection-sort) - [Insertion Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-insertion-sort) - **Merge Sort** - Quick Sort - Heap Sort As promised, we are back to explore the next in a series of sorting algorithms, the Merge Sort. Till now, we have reviewed three sorting algorithms as listed above, to conclude that the common traits they possess are inefficiency and slowness. Bubble Sort, Selection Sort, and Insertion Sort algorithms have minor differences among themselves with the same quadratic running time; meaning the time complexity of these three algorithms is O(n^2). In this post, we shall determine if Merge Sort is any better than its already discussed peers, and does it fit the bill to be chosen as "The One"? Let’s find out. ## Why is it called Merge Sort? Merge Sort is a[divide and conquer algorithm](https://en.wikipedia.org/wiki/Divide_and_conquer_algorithm) that was invented in 1945 by [John von Neumann](https://en.wikipedia.org/wiki/John_von_Neumann), who was a founding figure in computing. The Merge Sort algorithm sorts an input collection by breaking it into two halves. It then sorts those two halves and merges them together as one sorted array. Most of the merge sort implementations use divide and conquer, a common algorithm design paradigm based on [recursion](https://en.wikipedia.org/wiki/Recursion_(computer_science)). The idea behind the merge sort is, it’s easier to sort sublists or smaller sets and combine them rather than sorting the long list of an array’s items. In our [previous post for Insertion Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-insertion-sort), we have clearly identified the common issue in all the discussed sorting techniques as sorting a long list of values that had slower and inefficient runtimes, with a quadratic time complexity O(n^2). Let’s first understand the divide and conquer algorithm. > “*A divide-and-conquer algorithm works by recursively breaking down a problem into two or more sub-problems of the same or related type until these become simple enough to be solved directly. The solutions to the sub-problems are then combined to give a solution to the original problem.*” [[Ref. 1](https://en.wikipedia.org/wiki/Divide-and-conquer_algorithm)] A divide-and-conquer algorithm divides the problem into the simplest form. The smaller problems are the ones most easier to solve. The solution is then applied to the bigger chunks of the complex problem. Hence, the larger problem is conquered by using the same solution recursively. The three parts of the divide and conquer algorithm are as below: **Divide**: The problem is divided into the smallest possible number of the same problem; meaning the problem is divided into a subproblem, which indeed is divided into another subproblem and so forth until the smallest set is achieved. **Conquer**: The smallest subproblem is conquered first by finding the solution as a base case. Once the solution is achieved, the same technique can be used to tackle bigger subproblems recursively. **Combine**: The solution to the small problem is combined and build-up to solve the bigger problem. The below pictures demonstrate the divide-and-conquer algorithm, wherein a problem is divided into two subproblems and the algorithm makes multiple recursive calls. ![Fig1.1 Divide-and-Conquer demonstration - Khan academy [Ref. 2]](../../assets/blog/reviewing-sorting-algorithms-merge-sort-1.png) Further simplification of the recursive steps is demonstrated below: ![Fig1.2 Divide-and-Conquer demonstration - Khan academy [Ref. 2]](../../assets/blog/reviewing-sorting-algorithms-merge-sort-2.png) At this point, we have better clarity on how the Divide-and-conquer algorithm works in theory, it’s time to illustrate this design paradigm implementation in the Merge Sort. The idea is to discuss the Merge Sort in a very simple way so that it sticks well in our memory, very much like solving small problems to achieve the original big problem solution. We have now cemented in theory how the Merge Sort will split the collections in a number of sublists to find the basic solution. Let’s break it down with our post series sample array [10,8,4,6,1,3,2,5]. Additional element number [3,2,5] is added to the array to make it an even array for easier demonstration purposes. **Divide**: The array is divided into two halves initially and each half is again divided into halves until the smallest subset is attained. Below is the illustration for the divide step and we achieve a sorted smallest sub list at the end. You may ask, how is it sorted? Well, the final eight sublists are considered sorted as there is only one value and nothing else to compare. In short, the smallest sublist is always a sorted list by itself and considered as a base case. Now that we have solved the smallest problem, the next step would be to solve the next biggest subproblem. How do we do that? This is the time we introduce the “conquer” part of the algorithm. ![Merge sort divide step splitting an 8-element array into eight single-element sublists](../../assets/blog/reviewing-sorting-algorithms-merge-sort-3.png) **Conquer**: The sorted sublists are merged together maintaining the sorted list. Two sorted lists are required to merge together and create one single sorted list. The sorted lists are merged together as a part of the “*combine*” step recursively building towards the final single sorted list. The base case is the first single element sorted list. If you observe closely, the sorting has taken place recursively for each of the below sublists. ![Merge sort conquer step merging sublists back into one fully sorted array](../../assets/blog/reviewing-sorting-algorithms-merge-sort-4.png) We discussed the Divide-and-conquer algorithm on how it’s implemented in the Merge Sort and tried to simplify the workings of the Merge Sort, in theory. Let’s dive into code snippets to understand how it’s implemented in real-time. *Okay, but what about recursion?* We have used the reference multiple times in our discussion. [Recursion](https://en.wikipedia.org/wiki/Recursion_(computer_science)) is a method to solve the same problem by breaking it down into smaller problems until a base condition (ex: if n<= 1) is satisfied to stop the recursion. A recursive function is one that calls itself during execution directly or indirectly to achieve the desired solution. ## Pseudocode logic: - The recursive approach is followed to implement the Merge Sort. The first step is to find the mid-position of the array to split the array into two halves. Mid position for an array of ‘n’ size is found as ‘n/2’. Alternatively, (L+R)/2 would also yield the mid-position of the array where L is the left index, and R is the right index value. - If the input array length is less than 2 then no action will be performed, as this will be considered as a sorted single element sublist. - The array is split into two sublists of size ‘n/2’ until no further division can be performed. The split function ‘mergesort’ is invoked recursively to get the left and right sublist until a single element list is formed. - For each split function, ‘mergesort’ will invoke the function ‘merge’ each time recursively. For example, array [10,8,4,6,1,3,2,5] can be divided into two, four, and eight sublists, making it *mergesort(left), mergesort(right)* function call. - Once the single sublists are formed, the next step is to invoke the function ‘merge’ passing the input parameters left, right, and array. The single sublist is considered as a sorted list as there are no other values to compare, making it as our base case. - The logic of function *“merge“* implements the sorting and merging of the sublists. For example, single arrays [10] [8] are sorted and merged together as [8,10]. Similarly, backtracking other array values we attain the final sorted list. - The elements from the left array and right array indexes will be compared for sorting. In our example, the left array [8,10] and right array [4,6] the index[0] in both lists are compared to find the sorted position for the new sublist as [4,6,8,10]. ## Code Implementation: ```python #function for merging the left and right array. Sorting will be performed def merge(left, right, array): print(" ") print(" Merge -> array : {} Left : {} Right : {} ".format(array,left,right)) # Iterator i for traversing through the Left Half of the array i = 0 # Iterator j for traversing through the Right Half of the array j = 0 # Iterator k for the main list k = 0 while (i> Merged array is : {} ".format(array)); print(" ") return array ###################################################### ### Function for dividing and calling merge function # ###################################################### def mergesort(array): arrlen = len(array) if(arrlen<2): print(" array has only one element {} - no further action ".format(array)) return # derive the midpoint of the array mid = int(arrlen/2) # divide the array into left Half and Right Half based on the midpoint left = array[:mid] right = array[mid:] print(" Left Half :{} Right Half : {}".format(left,right)) print(" ") print(" Recursive call mergesort for left array {} ".format(left)) print(" ---------------------------------------------------------") mergesort(left) print(" ----------------------------------------------------------") print(" ") print(" Recursive call mergesort for Right array {} ".format(right)) print(" ----------------------------------------------------------") mergesort(right) print(" ----------------------------------------------------------") # print( " After MergeSort Right :{} ".format(right)) merge(left,right,array) return array # test driver code if __name__ == '__main__': # Test Long unsorted list # arrlist = [3, 56, 2, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] # arrlist = [] # Test short unsorted list arrlist = [10,4,8,6,1,3,2,5] #Test sorted list # arrlist = [1,2,3,4,5] #Test n= 100 unsorted list #arrlist = [52,68,84,73,3,81,91,58,1,19,8,78,32,94,4,98,23,6,21,20,43,69,31,87,33,62,36,41,70,51,48,38,72,24,47,57,13,7,64,49,9,16,14,99,90,92,35,25,37,89,50,85,76,67,88,131,39,56,74,42,77,26,93,97,80,82,96,60,53,46,18,65,30,27,2,83,66,29,61,10,55,22,11,75,95,45,59,86,12,15,17,79,40,34,5,71,44,63,28,54] print(" Input array : {}".format(arrlist)) print(" Output array :{} ".format(mergesort(arrlist))) ``` **Log Output:** ```text ############# Test Case 1 ############# Input array : [10, 4, 8, 6, 1, 3, 2, 5] Left Half :[10, 4, 8, 6] Rigth Half : [1, 3, 2, 5] Recursive call mergesort for left array [10, 4, 8, 6] --------------------------------------------------------- Left Half :[10, 4] Rigth Half : [8, 6] Recursive call mergesort for left array [10, 4] --------------------------------------------------------- Left Half :[10] Rigth Half : [4] Recursive call mergesort for left array [10] --------------------------------------------------------- array has only one element [10] - no further action ---------------------------------------------------------- Recursive call mergesort for Right array [4] ---------------------------------------------------------- array has only one element [4] - no further action ---------------------------------------------------------- Merge -> array : [10, 4] Left : [10] Right : [4] *-*-* End of merge function >> Merged array is : [4, 10] ---------------------------------------------------------- Recursive call mergesort for Right array [8, 6] ---------------------------------------------------------- Left Half :[8] Rigth Half : [6] Recursive call mergesort for left array [8] --------------------------------------------------------- array has only one element [8] - no further action ---------------------------------------------------------- Recursive call mergesort for Right array [6] ---------------------------------------------------------- array has only one element [6] - no further action ---------------------------------------------------------- Merge -> array : [8, 6] Left : [8] Right : [6] *-*-* End of merge function >> Merged array is : [6, 8] ---------------------------------------------------------- Merge -> array : [10, 4, 8, 6] Left : [4, 10] Right : [6, 8] *-*-* End of merge function >> Merged array is : [4, 6, 8, 10] ---------------------------------------------------------- Recursive call mergesort for Right array [1, 3, 2, 5] ---------------------------------------------------------- Left Half :[1, 3] Rigth Half : [2, 5] Recursive call mergesort for left array [1, 3] --------------------------------------------------------- Left Half :[1] Rigth Half : [3] Recursive call mergesort for left array [1] --------------------------------------------------------- array has only one element [1] - no further action ---------------------------------------------------------- Recursive call mergesort for Right array [3] ---------------------------------------------------------- array has only one element [3] - no further action ---------------------------------------------------------- Merge -> array : [1, 3] Left : [1] Right : [3] *-*-* End of merge function >> Merged array is : [1, 3] ---------------------------------------------------------- Recursive call mergesort for Right array [2, 5] ---------------------------------------------------------- Left Half :[2] Rigth Half : [5] Recursive call mergesort for left array [2] --------------------------------------------------------- array has only one element [2] - no further action ---------------------------------------------------------- Recursive call mergesort for Right array [5] ---------------------------------------------------------- array has only one element [5] - no further action ---------------------------------------------------------- Merge -> array : [2, 5] Left : [2] Right : [5] *-*-* End of merge function >> Merged array is : [2, 5] ---------------------------------------------------------- Merge -> array : [1, 3, 2, 5] Left : [1, 3] Right : [2, 5] *-*-* End of merge function >> Merged array is : [1, 2, 3, 5] ---------------------------------------------------------- Merge -> array : [10, 4, 8, 6, 1, 3, 2, 5] Left : [4, 6, 8, 10] Right : [1, 2, 3, 5] *-*-* End of merge function >> Merged array is : [1, 2, 3, 4, 5, 6, 8, 10] Output array :[1, 2, 3, 4, 5, 6, 8, 10] ############# Test Case 2 ############# Input array : [3, 56, 2, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] Left Half :[3, 56, 2, 58, 79, 59] Right Half : [34, 23, 4, 78, 8, 123, 45] >>> Recursive call mergesort for left array [3, 56, 2, 58, 79, 59] Left Half :[3, 56, 2] Right Half : [58, 79, 59] >>> Recursive call mergesort for left array [3, 56, 2] Left Half :[3] Right Half : [56, 2] >>> Recursive call mergesort for left array [3] Array has only one element [3] - no further action ---------------------------------------------------------- >>> Recursive call mergesort for Right array [56, 2] Left Half :[56] Right Half : [2] >>> Recursive call mergesort for left array [56] Array has only one element [56] - no further action ---------------------------------------------------------- >>> Recursive call mergesort for Right array [2] Array has only one element [2] - no further action ---------------------------------------------------------- Merge -> Array : [56, 2] Left : [56] Right : [2] *-*-* End of merge function ** array is : [2, 56] ---------------------------------------------------------- Merge -> Array : [3, 56, 2] Left : [3] Right : [2, 56] *-*-* End of merge function ** array is : [2, 3, 56] ---------------------------------------------------------- >>> Recursive call mergesort for Right array [58, 79, 59] Left Half :[58] Right Half : [79, 59] >>> Recursive call mergesort for left array [58] Array has only one element [58] - no further action ---------------------------------------------------------- >>> Recursive call mergesort for Right array [79, 59] Left Half :[79] Right Half : [59] >>> Recursive call mergesort for left array [79] Array has only one element [79] - no further action ---------------------------------------------------------- >>> Recursive call mergesort for Right array [59] Array has only one element [59] - no further action ---------------------------------------------------------- Merge -> Array : [79, 59] Left : [79] Right : [59] *-*-* End of merge function ** array is : [59, 79] ---------------------------------------------------------- Merge -> Array : [58, 79, 59] Left : [58] Right : [59, 79] *-*-* End of merge function ** array is : [58, 59, 79] ---------------------------------------------------------- Merge -> Array : [3, 56, 2, 58, 79, 59] Left : [2, 3, 56] Right : [58, 59, 79] *-*-* End of merge function ** array is : [2, 3, 56, 58, 59, 79] ---------------------------------------------------------- >>> Recursive call mergesort for Right array [34, 23, 4, 78, 8, 123, 45] Left Half :[34, 23, 4] Right Half : [78, 8, 123, 45] >>> Recursive call mergesort for left array [34, 23, 4] Left Half :[34] Right Half : [23, 4] >>> Recursive call mergesort for left array [34] Array has only one element [34] - no further action ---------------------------------------------------------- >>> Recursive call mergesort for Right array [23, 4] Left Half :[23] Right Half : [4] >>> Recursive call mergesort for left array [23] Array has only one element [23] - no further action ---------------------------------------------------------- >>> Recursive call mergesort for Right array [4] Array has only one element [4] - no further action ---------------------------------------------------------- Merge -> Array : [23, 4] Left : [23] Right : [4] *-*-* End of merge function ** array is : [4, 23] ---------------------------------------------------------- Merge -> Array : [34, 23, 4] Left : [34] Right : [4, 23] *-*-* End of merge function ** array is : [4, 23, 34] ---------------------------------------------------------- >>> Recursive call mergesort for Right array [78, 8, 123, 45] Left Half :[78, 8] Right Half : [123, 45] >>> Recursive call mergesort for left array [78, 8] Left Half :[78] Right Half : [8] >>> Recursive call mergesort for left array [78] Array has only one element [78] - no further action ---------------------------------------------------------- >>> Recursive call mergesort for Right array [8] Array has only one element [8] - no further action ---------------------------------------------------------- Merge -> Array : [78, 8] Left : [78] Right : [8] *-*-* End of merge function ** array is : [8, 78] ---------------------------------------------------------- >>> Recursive call mergesort for Right array [123, 45] Left Half :[123] Right Half : [45] >>> Recursive call mergesort for left array [123] Array has only one element [123] - no further action ---------------------------------------------------------- >>> Recursive call mergesort for Right array [45] Array has only one element [45] - no further action ---------------------------------------------------------- Merge -> Array : [123, 45] Left : [123] Right : [45] *-*-* End of merge function ** array is : [45, 123] ---------------------------------------------------------- Merge -> Array : [78, 8, 123, 45] Left : [8, 78] Right : [45, 123] *-*-* End of merge function ** array is : [8, 45, 78, 123] ---------------------------------------------------------- Merge -> Array : [34, 23, 4, 78, 8, 123, 45] Left : [4, 23, 34] Right : [8, 45, 78, 123] *-*-* End of merge function ** array is : [4, 8, 23, 34, 45, 78, 123] ---------------------------------------------------------- Merge -> Array : [3, 56, 2, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] Left : [2, 3, 56, 58, 59, 79] Right : [4, 8, 23, 34, 45, 78, 123] *-*-* End of merge function ** array is : [2, 3, 4, 8, 23, 34, 45, 56, 58, 59, 78, 79, 123] Output array :[2, 3, 4, 8, 23, 34, 45, 56, 58, 59, 78, 79, 123] ############# Test Case 3 ############# Input array : [1, 2, 3, 4, 5] Left Half :[1, 2] Right Half : [3, 4, 5] >>> Recursive call mergesort for left array [1, 2] Left Half :[1] Right Half : [2] >>> Recursive call mergesort for left array [1] Array has only one element [1] - no further action ---------------------------------------------------------- >>> Recursive call mergesort for Right array [2] Array has only one element [2] - no further action ---------------------------------------------------------- Merge -> Array : [1, 2] Left : [1] Right : [2] *-*-* End of merge function ** array is : [1, 2] ---------------------------------------------------------- >>> Recursive call mergesort for Right array [3, 4, 5] Left Half :[3] Right Half : [4, 5] >>> Recursive call mergesort for left array [3] Array has only one element [3] - no further action ---------------------------------------------------------- >>> Recursive call mergesort for Right array [4, 5] Left Half :[4] Right Half : [5] >>> Recursive call mergesort for left array [4] Array has only one element [4] - no further action ---------------------------------------------------------- >>> Recursive call mergesort for Right array [5] Array has only one element [5] - no further action ---------------------------------------------------------- Merge -> Array : [4, 5] Left : [4] Right : [5] *-*-* End of merge function ** array is : [4, 5] ---------------------------------------------------------- Merge -> Array : [3, 4, 5] Left : [3] Right : [4, 5] *-*-* End of merge function ** array is : [3, 4, 5] ---------------------------------------------------------- Merge -> Array : [1, 2, 3, 4, 5] Left : [1, 2] Right : [3, 4, 5] *-*-* End of merge function ** array is : [1, 2, 3, 4, 5] Output array :[1, 2, 3, 4, 5] ``` **Observations based on code implementation:** - We defined two functions *“merge”* and *“mergesort”* to implement the Divide-and-conquer algorithm. From the logs, we see that the array [10,4,8,6,1,3,2,5] is divided into Left Half: [10, 4, 8,6] Right Half: [1,3,2,5]. - To further divide the arrays, we see the recursive algorithm in action by invoking function “mergesort” multiple times. As shown in the logs, the first set of [10,4,8,6] is divided until the single element, further sorting and merging it as [4,6,8,10]. In the same way, the right array is also sorted and built-up to merge and form a new array [1,2,3,5]. - Merge Sort performs three steps: dividing the collection, sorting the values, and merging it back together. Multiple sublists are sorted at a time in the Merge Sort due to recursion, whereas in case of iteration we sort one item at a time. This is a clear distinction from its peers Bubble Sort, Selection Sort, and Insertion Sort. The recursive action is more efficient than iterations, as the work is constantly divided into halves. - The function *“merge”* is the one that does the heavy lifting by sorting and merging back the arrays. In short, we are appending the items from a single element to form the complete array as a new array structure in sorted order. The step *“merge”* actively compares two sublists and then appends the smaller value as the first index and larger value at the second index position. - As shown below, in total we must perform n= 8 and 3 recursion level operations for merge and sorting. In each level, we will be performing sorting and merge of the elements to store it in a new temporary sublist array. As there are 3 levels we have performed a total of 24 operations of comparing the elements and inserting them in a new array. Remember, the length of our array is 8, how many recursion levels will we have for the size n = 16 array? It would be 4 recursion levels, similarly, for a size ‘n’ number of an array, the recursion level will be the logarithmic value of Log2(n). - From the above abstraction, in case n = 8 (array size) multiplying with Log2(8) value (i.e 3) will give 24 as the number of operations to be performed. For time complexity, it is read as O(n*log(n)) time taken for the operations to be performed for the Merge Sort. ![Recursion tree and operation counts illustrating merge sort's O(n log n) complexity](../../assets/blog/reviewing-sorting-algorithms-merge-sort-5.png) - The code implementation log has three test cases, out of which one case is for a completely sorted array. The sorted array will also go through the discussed Divide-and-conquer algorithm process and comparisons are performed to find the rightful sorting position. This indicates there is no distinction in runtimes for both the best-case scenarios of sorted arrays and an unsorted array. ## Time and space complexity - As we see from the above illustration, as the size of ‘n’ increases, so does the number of merge operations required. if we recollect from our previous posts, Bubble Sort, Insertion Sort, and Selection Sort all have worst-case time complexity is O(n^2) as the size of ‘n’ increases, the runtime is quadratic (n(n-1)/2) times. Whereas in the case of the Merge Sort runtime is not quadratic. - Merge Sort’s time complexity follows both linear and logarithmic runtime, where O(n) is linear and O(log N) is logarithmic. The division of the “mergesort” function indicates logarithmic time complexity, whereas the time complexity of the “merge” function and recursion is linear indicating O(n) time complexity. The merge sort combining both linear and logarithmic runtimes follows a time complexity called “linearithmic time” which is O(n log n). - In the case of Merge Sort, the time complexity for the best case, worst case, and average case scenarios are the same as O(*n* log *n*). - It’s time to refer to the Big-O Complexity chart we referred back in our [first post series of the Bubble Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-bubble-sort). From the below chat we can see that Merge Sort performs a lot better than its peers with quadratic runtimes O(n^2). ![Fig3.1 Big-O Algorithm Complexity Chart [Ref.4]](../../assets/blog/reviewing-sorting-algorithms-merge-sort-6.png) - We have discussed stability in our [previous post](https://www.prostdev.com/post/reviewing-sorting-algorithms-insertion-sort), the Merge Sort is a stable algorithm as it maintains the stability of the input data set. - As the Merge Sort requires a temporary array to store the sorted merge values, it requires an O(n) or constant amount of space. This is the memory needed for the temporary buffer array storage. The Merge Sort is an out-of-place sorting algorithm, which requires additional memory as its dataset grows. The Bubble Sort, Insertion Sort, and Selection Sort are in-place algorithms. > “*An in-place algorithm is an* [algorithm](https://en.wikipedia.org/wiki/Algorithm) *that transforms input using no auxiliary* [data structure](https://en.wikipedia.org/wiki/Data_structure)*. However, a small amount of extra storage space is allowed for auxiliary variables. The input is usually overwritten by the output as the algorithm executes. In-place algorithm updates the input sequence only through replacement or swapping of elements. An algorithm that is not in-place is sometimes called not-in-place or out-of-place."* [[Ref. 3](https://en.wikipedia.org/wiki/In-place_algorithm)] **Time and space complexity table:** ![Table comparing time and space complexity of merge, insertion, selection, and bubble sort](../../assets/blog/reviewing-sorting-algorithms-merge-sort-7.png) ## Final thoughts The question is, did we find “The One”? We certainly see some potential in the Merge Sort as a faster and more efficient sorting algorithm with minor drawbacks, compared to the most popular kid Bubble Sort, followed by Selection and Insertion Sort. - In the case of large data sets, the Merge Sort is definitely way faster than its peers O(n^2) with a linearithmic time complexity of O(*n* log *n*) in best, average, and worst case scenarios. - However, the time complexity of the Insertion Sort’s best case scenario is O(n), which is better than the Merge Sort in case of a fully sorted array. - The space complexity of the Insertion Sort O(1) is better than the Merge Sort’s O(n) as it requires additional space for a temporary buffer array in case of large data sets. This is one of the drawbacks of the Merge Sort being an out-of-place algorithm requiring more memory as the dataset grows. Considering the above drawbacks, for small datasets it will be beneficial to use the Insertion Sort and for large datasets, the best suited so far is the Merge Sort. In fact, in Python and Java, the sort function is implemented as a hybrid of the Merge Sort and Insertion Sort called [Tim Sort](https://en.wikipedia.org/wiki/Timsort). For hybrid models, the Insertion Sort is used for the sublist of the Merge Sort as it runs faster for small data sets and has efficient space complexity. The hybrid model can be considered as “The One”, but hey, there are few other candidates auditioning for the title. The next candidate up for auditioning is known to quickly sort the data set using the “Quick Sort” algorithm. Stay tuned!! I hope you enjoyed the post. Please subscribe to[ProstDev](https://www.prostdev.com/) for more exciting topics. Hasta luego, amigos! ## References - [Divide-and-conquer algorithm - Wikipedia](https://en.wikipedia.org/wiki/Divide-and-conquer_algorithm) - [Divide-and-conquer algorithm - Khan Academy](https://www.khanacademy.org/computing/computer-science/algorithms/merge-sort/a/divide-and-conquer-algorithms) - [In-place algorithm - Wikipedia](https://en.wikipedia.org/wiki/In-place_algorithm) - [Big-O Algorithm Complexity Chart](https://www.bigocheatsheet.com/) ## Study Resources - [Recursion - Khan Academy](https://www.khanacademy.org/computing/computer-science/algorithms/recursive-algorithms/a/recursion) - [Video] [Recursion - CS50 Stanford](https://www.youtube.com/watch?v=mz6tAJMVmfM) - [Differentiating Logarthamic and linearithmic time complexity](https://levelup.gitconnected.com/differentiating-logarithmic-and-linearithmic-time-complexity-976cd49c351b) - [Analysis of Merge Sort - Khan Academy](https://www.khanacademy.org/computing/computer-science/algorithms/merge-sort/a/analysis-of-merge-sort) - [Logarithm Calculator - RapidTable](https://www.rapidtables.com/calc/math/Log_Calculator.html) - [Video] [Merge Sort with Transylvanian German Folk Dance](https://www.youtube.com/watch?v=HzUrYDdXY-8) - [Video] [CS50 - Merge Sort](https://www.youtube.com/watch?v=Pr2Jf83_kG0) - [Merge Sort - GeekforGeeks](https://www.geeksforgeeks.org/merge-sort/) --- ## IoT with MuleSoft: Implementing a Temperature Sensor using LED lights, Twilio, and a Raspberry PI Source: https://prostdev.com/post/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi | Published: Aug 20, 2020 | Category: Tutorials Hey Muleys, In this post, I will explain how to implement IoT with a MuleSoft use case. I would like you to try out some new use cases and publish them. I hope this article helps you to implement them! First of all, why should you use MuleSoft to implement IoT? Why not use other integration platforms? Quick answer: Ease of Integration and Speed of Delivery. Let's dive in. Make sure to watch the videos posted at the end of the article! You will get a full idea on how easy it is to implement IoT with MuleSoft. ## What is IoT? The Internet of Things (IoT) is the inter-network of physical devices, vehicles, buildings, and other items embedded with electronics, software, sensors, actuators, and network connectivity that enable these objects to collect and exchange data. Simplified: IoT is about connecting software with hardware! **MuleSoft + IoT:** - The Mule engine can be embedded directly into IoT devices, which enables data exchange for the devices by connecting to IoT cloud services and backend apps in the cloud. - The Mule Runtime engine can be used to expose APIs on any IoT device. Mule APIs can be deployed on IoT devices and turn them on and off. - In this article we will discuss about IoT and how it can be used with MuleSoft, and how Mule APIs can be deployed on IoT devices. ## Use Case When a user passes a receiver’s number in the URL, the receiver should read the current local temperature details to his mobile and on successful receiving of details on his mobile, a green LED light should be blinked. In case of any issue in receiving the details, the red LED light should be blinked instead. **How does it work internally?** When you hit this endpoint: ``` http://localhost:8081/test?toNumber=919999999999 ``` - The request is sent to the Mule application. - The receiver’s number is stored in a variable. - The Mule application connects to Raspberry PI (IoT device) and senses the temperature using a temperature sensor. - The temperature is stored in a variable. - The Mule application sends the details to the receiver’s mobile number using the Twilio Connector. - If the data is received as expected, a green LED light is blinked, otherwise it will be a red LED light. ## Let's cook the recipe **Required Software:** - [Raspberry Pi OS (previously called Raspbian OS)](https://www.raspberrypi.org/downloads/raspbian/) - [SD Memory Card Formatter](https://www.sdcard.org/downloads/formatter/) - [Xming Display Server](https://xming.en.softonic.com/) - [Win32 Disk Imager](https://win32diskimager.download/) - [PuTTY](https://www.putty.org/) - [WinSCP](https://winscp.net/eng/download.php) - [Mule Standalone Server](https://www.mulesoft.com/lp/dl/mule-esb-enterprise) **Required Hardware:** - Raspberry Pi 3 - Micro SD Card of 16GB - Ethernet Cable or HDMI cable (in this use case we are using ethernet cable) - Adapter Charger for Raspberry Pi 3: Available in electronic stores or any e-commerce websites as a combo. I was able to find everything [here](https://www.amazon.in/gp/product/B07C6SN8PL/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1). **Resistors:** - 4k7 ohms (4.7k) - 1 (used for temp sensor) - 22k ohms: 2 in total (used for RED and GREEN LED lights) - Jumper wires: Male – Female (10 for safe side) - 1 Bread Board - Temperature Sensor: DS18B20 model - 1 red LED light - 1 green LED light ## Part 1 - Setting up your environment Before starting with the Raspberry PI setup, we have to follow these steps. **Step 1: Download and Install Raspberry Pi OS / Raspbian OS** Download Raspbian OS (URL can be found in Required Software above). Select “Raspbian Buster with desktop and recommended software”. It’s almost 2.5 GB. ![Raspberry Pi download page for Raspbian Buster with desktop and recommended software](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-1.png) While it’s downloading, insert your micro SD card into your system and format it using the SD Card formatter. Now extract the downloaded Raspbian OS into your local. You can see only one file of the type Disk Image. We can’t unzip them normally. That’s the reason we use Win32 disk imager: to extract and copy them to the Micro SD card. Open the Win32 disk imager and you will see that the destination folder will be automatically detected (the SD card). Just select the file where you have extracted Raspbian and click on write. It takes around 9 mins to finish. Once it’s done it will say that the writing was successful. Now your OS is copied successfully in your SD card. **Step 2: Enable SSH** The SSH command provides a secure encrypted connection between two hosts over an insecure network. This connection can also be used for terminal access, file transfers, and for tunneling other applications. Since we need to see what’s happening in the Raspberry PI, we will need a UI to see this. In order to set this up, we need to have SSH enabled. Go to the SD card folder where you can see the extracted files (they were extracted using Win32 disk imager). Now just create a text file and name it as ssh. After creating this, you can remove the SD Card from your PC. **Step 3: Network and Sharing** Now we need to connect our Raspberry device to our system. As I said before, we have 2 ways. One is using an HDMI cable which requires WiFi sharing, a monitor, a keyboard and a mouse to perform operations. To go for the alternative, it’s better to use an ethernet cable which helps us to get connected with our PC itself. For this article, we are using ethernet connectivity. After ejecting the SD card from the PC, mount the SD card to your Raspberry PI. See the slot where it needs to be inserted (usually it will be on the downside of the device). Plug the adaptor and switch it on. Now connect the ethernet cable: one side should be connected to the Raspberry PI and the other to your PC. Make sure your PC is already connected to WiFi or internet before plugging the ethernet cable. Once the ethernet cable is plugged to your PC. Go to the Network and Sharing center. You will see an unidentified network. You can refer to the next screenshot. ![Windows Network and Sharing Center with an Unidentified Ethernet network circled](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-2.png) Click on the internet connection which you have already connected to and click on WiFi – properties. Go to Sharing and make sure the 2 options in Internet Connection Sharing are checked. Also see that Home networking connection is automatically generated with the name “Ethernet” (you can go through the video I shared at the end to verify these steps). ![Wi-Fi Properties Sharing tab with Internet Connection Sharing enabled for Ethernet](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-3.png) Click on unidentified network (ethernet). Go to Properties and double click on Internet Protocol Version 4 (TCP/IPv4). Under “Use the following IP address” see that an IP address is automatically shown. Copy that IP address as we will be using the same IP to connect to the Raspberry PI. ![IPv4 Properties showing the auto-assigned IP address 192.168.137.1](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-4.png) **Step 4: Connect to the Raspberry PI using PuTTY** Before connecting using PuTTY, make sure you have open Xming (assuming it's already been downloaded and installed). You can open PuTTY after this. Include the following configuration: Hostname: `raspberrypi.mshome.net` (this is the default hostname for Raspberry Pi. You can also use the IP address generated above, but it’s better to use the mentioned hostname) Port: 22 Go to SSH and include the following configuration: ![PuTTY SSH X11 settings with Enable X11 forwarding checked and display set to localhost:0](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-5.png) Save with any name. Now you’re ready to connect to the device. Click on Open. You will get a security alert warning. Click on yes to proceed. A cmd prompt is popped out asking to login. You can select the default login details like the following: Login as: pi Password: raspberry Now you are connected to the “pi” user of your Raspberry device. To see a graphical view of your device (generally another system which runs on Raspbian OS) you can use the below command in PuTTY command: ```bash pi@raspberrypi : $ startlxde ``` Now your Xming opens! ![Raspberry Pi LXDE desktop shown through Xming with an LXTerminal icon](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-6.png) Leave your LXTerminal pinned on your desktop. Open the Terminal and type the below command: ```bash pi@raspberrypi : $ sudo raspi-config ``` A blue prompt appears. It has option1 to change password. Make sure you change the password and finish off. **Step 5: Install Java 8** We need to install Java version 8. By default, Java and Python are already installed when we download Raspbian OS. If you check the version (using the java -version command), we can see the Java 11.x version is already installed. We have some permission issues with this version. This is why we need to install Java 8 for this use case. Before installing, you will need to uninstall Java 11. Use below commands for uninstalling java 11 and then installing java 8 accordingly. To uninstall Java 11: ```bash pi@raspberrypi : $ sudo apt-get purge openjdk* ``` To install Java 8: ```bash pi@raspberrypi : $ sudo apt-get install openjdk-8-jdk ``` Now check the Java version again. It will show Java 8. We have set up our Raspberry OS. All 5 steps are related to Raspberry and have nothing to do with Mule installation. These are common steps for setting up a Raspberry PI device. ## Part 2 - Setting up Raspberry PI with Mule **Step 1: Installing Mule Standalone Server** Open a browser in Xming (in the Raspberry PI). Go to the link I shared in the Required Software to download Mule Standalone Server. While it’s downloading, we shall create a new user with the name “mule”. All MuleSoft operations are carried out by this user (this is not mandatory but it’s good to create one). To create a Mule user type the following commands: ```bash pi@raspberrypi : $ sudo su – root@raspberrypi : # useradd -s /bin/bash -d /home/mule -U -G sudo mule root@raspberrypi : # passwd mule New Password: Retype New Password: ``` Your “mule” user is now created successfully. Now you need to create a director and give all necessary permissions. For this, use the following commands: ```bash root@raspberrypi : # mkdir /home/mule /opt/mule root@raspberrypi : # chown mule:mule /opt/mule root@raspberrypi : # exit logout ``` Now it's time to look at the Mule Standalone Server. Your Mule Standalone will be located in the Downloads folder of the PI user. Follow these commands next: ```bash pi@raspberrypi : $ cd /home/pi/Downloads pi@raspberrypi :~/ Downloads$ chmod 777 * pi@raspberrypi :~/ Downloads$ su -mule Password: mule@raspberrypi :~$ cd /home/pi/Downloads mule@raspberrypi : /home/pi/Downloads$ cp mule-ee-distribution-standalone-4.2.2.zip /opt/mule mule@raspberrypi : /home/pi/Downloads$ cd /opt/mule mule@raspberrypi : /opt/mule$ unzip mule-ee-distribution-standalone-4.2.2.zip mule@raspberrypi : /opt/mule$ cd mule-ee-distribution-standalone-4.2.2 mule@raspberrypi : /opt/mule/mule-ee-distribution-standalone-4.2.2$ cd /opt/mule ``` Mule runtime uses the Tanuki Service Wrapper, which allows a Java-based application (that’s right, such as Mule runtime) to be started as a Windows Service or UNIX daemon. However, out-of-the-box, the bundled Service Wrapper is not optimized for Raspberry Pi’s ARM architecture. Therefore, the next step is to download the Armhf port of the Java Service Wrapper and patch the bundled Service Wrapper by copying a few required files to the Mule runtime directory. Additional config files needed: ```bash mule@raspberrypi:/opt/mule $ wget https://download.tanukisoftware.com/wrapper/3.5.34/wrapper-linux-armhf-32-3.5.34.tar.gz tar zxf wrapper-linux-armhf-32-3.5.34.tar.gz mule@raspberrypi:/opt/mule $ cp ./wrapper-linux-armhf-32-3.5.34/lib/libwrapper.so ./mule-standalone-4.2.2/lib/boot/libwrapper-linux-armhf-32.so mule@raspberrypi:/opt/mule $ cp ./wrapper-linux-armhf-32-3.5.34/lib/wrapper.jar ./mule-standalone-4.2.2/lib/boot/wrapper-3.2.3.jar mule@raspberrypi:/opt/mule $ cp ./wrapper-linux-armhf-32-3.5.34/bin/wrapper ./mule-standalone-4.2.2/lib/boot/exec/wrapper-linux-armhf-32 ``` Exit current terminal and re-open it. It comes with pi user ```bash mule@raspberrypi:/opt/mule $ cd mule-enterprise-standalone-4.2.2 mule@raspberrypi:/opt/mule/mule-enterprise-standalone-4.2.2$ cd conf mule@raspberrypi:/opt/mule/mule-enterprise-standalone-4.2.2/conf $ vi wrapper.conf ``` The mule file comes in edited format. Change below lines as mentioned and save it: ``` ## Initial Java Heap Size (in MB) wrapper.java.initmemory=256 ## Maximum Java Heap Size (in MB) wrapper.java.maxmemory=512 ``` Final step: ```bash mule@raspberrypi:/opt/mule/mule-enterprise-standalone-4.2.2 $ cd bin mule@raspberrypi:/opt/mule/mule-enterprise-standalone-4.2.2/bin $ ./mule ``` The server is up and running now: ![Mule standalone server console log showing the temperature app deployed and Mule running](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-7.png) **Step 2: Developing the Mule Application** Now we shall develop a RESTful application in your local (Windows) system. And then run the application which generates a snapshot jar file. ![Anypoint Studio flow: Listener, transforms, Groovy scripts, Twilio Send message, and a try/error-handler](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-8.png) Connectors used: - Groovy script: This component is used to run python scripts. Use operation groovy. - Twilio: This component is used to send a message to the user. - Transform Message: Setting variables and structuring the message before sending to twilio. > [!NOTE] > You are not writing any Python scripts in the Mule application. The Python scripts to light the LEDs and sense temperature are already written and placed in a specified location. The location path is given in the groovy scripts. Make sure to give a proper message structure before sending to Twilio, since it expects a specific format containing these fields: body, to, from, message. Also make sure you have a developer account with Twilio and make sure whatever number you are using is already registered with Twilio. Messages are sent only to registered numbers. If you are not okay with using Twilio, you can build your own use case. Like printing the temperature instead. Code for the developed app is pasted at the end of this article. Run your application. Once it’s deployed locally, copy the jar file generated inside the target folder and paste it into your Raspberry device folder using WinSCP. To do this, open WinSCP, use the same host and port details from the ones you used for PuTTY. It will ask for username and password. Enter username as “pi” and the previously specified password. Copy the file from your Windows PC into the /home/pi/Downloads folder of your Raspberry PI. **Step 3: Connections in Raspberry PI** ![GPIO pinout chart beside a Raspberry Pi wired to a breadboard](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-9.png) Remember – Physical numbering is different from GPIO pin number. Watch the video for the details on how to create these connections. You can refer to the following pictures. ![Raspberry Pi GPIO pinout diagram mapping physical pin numbers to GPIO labels](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-10.png) ![Breadboard with a temperature sensor and green and red LEDs wired with jumper cables](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-11.png) Connection with Jumper wires: > [!NOTE] > physical numbering is normal 1,2,3…42 GPIO is different. For the connections please go with physical numbering. In the Python script you can see GPIO pin numbers. Do not get confused. **Green LED light:** GPIO Numbering: - Voltage: GPIO17 - Ground: Any nearest ground pin Physical Number: - Voltage: 11th pin - Ground: 14th pin **Red LED light:** GPIO Numbering: - Voltage: GIPO26 - Ground: Any nearest ground pin Physical Number: - Voltage: 37th pin - Ground: 39th pin **Temperature sensor:** GPIO Numbering: - Voltage: 3V3 - Ground: Any nearest ground pin - Data: GPIO 4 Physical Number: - Voltage: 1st pin - Ground: 6th pin - Data: 7th pin > [!IMPORTANT] > Before executing the Python scripts via Mule app, make sure you run the Python scripts independently (Python is already installed when Raspbian OS is installed). ```bash $ python /home/pi/Downloads workingTemp.py ``` Open LxTerminal: ```bash pi@raspberrypi: ~ $ cd /home/pi/Downloads pi@raspberrypi: ~/Downloads $ chmod 777 raspberry-temperature-new-1.0.0-SNAPSHOT-mule-application.jar pi@raspberrypi: ~/Downloads $ chmod a+x raspberry-temperature-new-1.0.0-SNAPSHOT-mule-application.jar ``` Write Python Scripts: greenLight.py: ![Terminal showing the greenLight.py Python script that toggles GPIO 17 high then low](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-12.png) redLight.py: ![Terminal showing the redLight.py Python script that toggles GPIO 26 high then low](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-13.png) Temperature.py: ![Terminal showing the workingTemp.py script reading the DS18B20 sensor and converting to Fahrenheit](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-14.png) Give permissions to Python scripts: ```bash pi@raspberrypi: ~/Downloads $ chmod 777 greenLight.py pi@raspberrypi: ~/Downloads $ chmod a+x greenLight.py pi@raspberrypi: ~/Downloads $ chmod 777 redLight.py pi@raspberrypi: ~/Downloads $ chmod a+x redLight.py pi@raspberrypi: ~/Downloads $ chmod 777 temp.py pi@raspberrypi: ~/Downloads $ chmod a+x temp.py ``` Final step: ```bash pi@raspberrypi: ~/Downloads $ su -mule Password: mule@raspberrypi: ~ $ cd /home/pi/Downloads mule@raspberrypi:cd /home/pi/Downloads$ cp raspberry-temperature-new-1.0.0-SNAPSHOT-mule-application.jar /opt/mule/mule-enterprise-standalone-4.2.2/apps ``` As the server is already up and running, the app is successfully deployed. ## Videos - For success: [Blink green LED on success](https://youtu.be/pEvFYxv8Hq0) - For failure: [Blink red LED on failure](https://youtu.be/HJGBqjTURco) - For connections on breadboard: [Raspberry Pi connection to breadboard](https://youtu.be/qTg3k8hxG6k) - For full setup: [Mule 4 standalone server on Raspberry Pi](https://youtu.be/TLOL9hwjNsY) ## Hardware Pictures ![Raspberry Pi 3 board with a yellow Ethernet cable plugged in](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-15.png) ![Hands holding two jumper wires with male and female ends](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-16.png) ![Hand holding a small DS18B20 temperature sensor with three leads](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-17.png) ![Hand holding a small resistor with its leads bent outward](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-18.png) ![Green LED and a resistor inserted into a breadboard](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-19.png) ![Raspberry Pi 3 with colored jumper wires connected to its GPIO header](../../assets/blog/iot-with-mulesoft-implementing-a-temperature-sensor-using-led-lights-twilio-and-a-raspberry-pi-20.png) ## Mule App Code ```xml def command = "python /home/pi/Downloads/workingTemp.py" println "$command" def cmd=command.execute() def command = "python /home/pi/Downloads/greenLight.py" println "$command" def cmd=command.execute() def command = "python /home/pi/Downloads/redLight.py" println "$command" def cmd=command.execute() ``` Hope this article helps you to do some real time use cases! Happy Learning!! Yours, Sravan Lingam :) --- ## Cryptography Module Mule 4 - Part 1 (PGP Encryption/Decryption) Source: https://prostdev.com/post/cryptography-module-mule-4-part-1-pgp-encryption-decryption | Published: Aug 18, 2020 | Category: Tutorials *GitHub repository with the Mule project can be found at the end of the post.* Mule 4 has a Cryptography module which includes these 3 different strategies: - PGP - XML - JCE In this article, we will see the PGP technique. ## PGP Pretty Good Privacy (PGP) is a cryptographic way that allows secure communication between two entities. It uses the public and private key concepts to encrypt the data as shown in the below diagram. ![PGP diagram: A encrypts a message with B's public key and B decrypts it with the private key](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-1.png) ## Prerequisites **1. Install the Crypto Module from Exchange, located in the Mule palette.** > [!NOTE] > Here is the reference documentation on how to install new modules to your Mule Project: [Adding Modules to Your Project](https://docs.mulesoft.com/studio/7.5/add-modules-in-studio-to). ![Mule Palette Crypto module listing PGP, JCE, and XML operations like Pgp encrypt and Pgp decrypt](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-2.png) **2. Create private and public keys** Please follow the below steps to generate a public/private key pair: - Download and install the GnuPG from [this link](https://www.gnupg.org/download/index.html). - Install [Kleopatra](https://www.openpgp.org/software/kleopatra/) to use a Graphical User Interface (GUI). - To generate the keys, click on “New Key Pair” and follow the instructions on the screen. ![Kleopatra welcome screen with New Key Pair and Import buttons for managing PGP keys](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-3.png) - Once the keys are generated, export them to the file system. ![Kleopatra context menu noting Export gives the public key and Export Secret Keys the private key](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-4.png) - The generated files are of ASC format, which is not supported by Mule yet, so we need to [dearmor](https://www.gnupg.org/documentation/manuals/gnupg/Operational-GPG-Commands.html) the keys first. Run the following command: `./gpg --dearmor ` for each of the keys. This command will create new files alongside the ASC files that will have .gpg appended to their filename which are supported in Mule. ![File listing with private.asc and public.asc next to their dearmored .gpg versions](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-5.png) This is what you will get after following the previous steps: - Public/Private keys - Fingerprint - Passphrase - KeyId ## Mule Code Implementation We will limit our scope to PGP encrypt/decrypt operation in this article. ### Global configurations Create 2 global configurations: - Encryption – Configure public key, keyId, and fingerprint. - Decryption – Configure private key, keyId, and fingerprint. ![**For projects: you should have these 2 configurations in different projects and all fields should be read from property files. The passphrase should be treated as a secure property.](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-6.png) ### Usage **1. Encryption/Decryption of entire payload** **Encryption:** ![**To log or use payload in the next processor, convert the encrypted stream to a base64 data type.](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-7.png) **Output:** ![Postman POST to /encrypt returning a PGP-encrypted payload as a base64 string](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-8.png) **Decryption:** ![Mule XML for the decryption flow converting the base64 payload to a stream and running Pgp decrypt](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-9.png) **Output:** ![Postman POST to /decrypt returning the original JSON payload with name, city, and email](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-10.png) **2. Encryption/Decryption at field level** This is a very common non-functional requirement where sensitive fields should be encrypted. For this, we can still reuse the pgp-encryption-flow and pgp-decryption-flow flows. The only change would be the way we refer to these flows from the main flow, and for that, the DataWeave lookup function is very useful. **Encryption:** ![**Used lookup function to invoke the pgp-encryption-flow.](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-11.png) **Output:** ![Postman POST to /encrypt returning JSON where only the EMAIL field is PGP-encrypted](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-12.png) **Decryption:** ![Mule XML using a DataWeave decrypt function with lookup to decrypt only the email field](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-13.png) **Output:** ![Postman POST to /decrypt returning JSON with the email field decrypted back to its original value](../../assets/blog/cryptography-module-mule-4-part-1-pgp-encryption-decryption-14.png) ## Conclusion So far we learned Mule code implementation for PGP. You can set this Mule code as a common service that will help to achieve the encryption/decryption non-functional requirement in many APIs. Here are the 2 ways to set up this Mule code as a common service: - Externalize the flow and publish it on the Anypoint Exchange. - Create a common API, which will encrypt/decrypt the payload. ## References - [https://docs.mulesoft.com/mule-runtime/4.3/cryptography-pgp](https://docs.mulesoft.com/mule-runtime/4.3/cryptography-pgp) - [https://docs.mulesoft.com/mule-runtime/4.3/dw-mule-functions-lookup](https://docs.mulesoft.com/mule-runtime/4.3/dw-mule-functions-lookup) ## GitHub repository [ProstDev GitHub - Demo PGP](https://github.com/ProstDev/demo-pgp) --- ## 3 simple rules & 5 examples to understand the Error Handling in Mule 4 Source: https://prostdev.com/post/3-simple-rules-5-examples-to-understand-the-error-handling-in-mule-4 | Published: Aug 13, 2020 | Category: Guides Hey Muleys, I believe that this article on Error Handling in Mule 4 will definitely help you to understand the core concept of On-Error Propagate and On-Error Continue. ## What is Error Handling? An exception occurs when an unexpected event happens while processing. Exception (or error) handling is the process of responding to exceptions when a computer program runs. In Mule, we can handle the message exception at different levels. - At project level using the **Default error handler** - At project level using the **Custom Global error handler** - At flow level in exception handling using the **Raise Error** component, **On-Error Continue** and **On-Error Propagate**. - At flow or at processor level using the **Try scope**. Whenever an error occurs in a flow, an error object is created. It contains many properties, like error.description or error.errorType. The errorType property is a combination of Namespace and Identifier. For example, HTTP:UNAUTHORIZED, where the Namespace is HTTP and the Identifier is UNAUTHORIZED. Mule identifies the error based on the errorType and then routes it to its respective block that is placed inside the Error Handler. ## What is On-Error Propagate and On-Error Continue? The difference between a flow, a sub-flow and a private flow, is that sub-flow doesn’t have an Error Handling scope. So, it's just flow and private flow the ones that have an Error Handling block and can contain the On-Error Propagate and On-Error Continue. > Whether it is Propagate or Continue, Mule Executes all the components within those blocks. **Remember:** The error will route to the Error Handling only if it identifies that the errorType from the error matches with what you have set in your Error Handling block. Mule 4 has one excellent feature that can identify the types of errors that can occur within that flow by looking at what kind of connectors are placed in that particular flow. ![Anypoint Studio error-type picker listing DB and HTTP error types available to the flow](../../assets/blog/3-simple-rules-5-examples-to-understand-the-error-handling-in-mule-4-2.png) You can see above that, since we have an HTTP Request and Database components, the drop-down shows all the types of errors that can happen in different scenarios from these two components. It even shows the EXPRESSION error type (not in the picture) because there are some DataWeave syntaxes in the flow. It will be very easy to learn Error Handling by knowing how On-Error Propagate and On-Error Continue work. We shall follow some set of rules to identify the flow of process. That way we can determine what is the result payload statusCode. ## Rules Before learning about rules. We must not forget that a RAML-based generated HTTP Listener will have the Error Response Body set as “payload” by default, but a manually drag-and-drop HTTP Listener will have output text/plain - - - error.description set by default. So, remember to change this according to your business requirements. Here are the rules to remember when an error has occurred in any flow: **Rule 1:** - If anything is present in the Error Handling section. Even if there are On-Error Propagate and On-Error Continue blocks, make sure that your particular errorType is handled by those. - If not, then Mule will use the default Error Handling. If your flow is not called by any other flow, then it will display the default value that is set in the Error Response Body and will return a status code of 500 by default, if nothing is manually set. If your flow is called by any other flow, then the error will be raised to the calling flow. **Rule 2:** If Error Handling is present in that particular flow and errorType is handled (Rule 1.1), then check whether it is On-Error Continue or On-Error Propagate. In both the cases, Mule will execute all components within that block. **Rule 3:** After the execution of all the components, now: - If the error is handled using On-Error Propagate, it will raise an error back to the calling flow. - If the error is handled using On-Error Continue, it will not raise an error back to the calling flow and continue to the next processor after “flow-ref” and continues further process as it is. But it will not continue to other processors in the flow where the error is handled. - Suppose you have only a single flow. Then On-Error Propagate and On-Error Continue behave the same way, except that On-Error Continue gives 200 status and On-Error Propagate gives 500 status. Because, as explained, even if it’s On-Error Continue, it will not go to the next processor within the same flow. - Always remember, this point is very important. **On-Error Continue will continue only to the next processor of the calling flow but it will not continue within the same flow.** If you observe the flows below, suppose your error is handled in the second private flow and if there is On-Error Continue, it will not execute the logger in private flow 2, instead, it will go to the Set Payload component of private flow 1, which is the one calling private flow 2. ![Two flows: flow 1 with On Error Propagate calling flow 2, which has a Logger and On Error Continue](../../assets/blog/3-simple-rules-5-examples-to-understand-the-error-handling-in-mule-4-3.png) If it is On-Error Propagate, the error is raised back to the calling flow. Now follow Rule 1 to Rule 3 again for that particular calling flow until the final process is completed. Whenever an error is raised back to the calling flow, just go by the three rules again. ## Examples Let’s see some examples and check what is the output and status code of each one. Consider that in all cases the default value for the Error Response in HTTP Listener is set to #[payload] and all cases below raise the same error for errorType: HTTP:CONNECTIVITY. All cases have the same input payload, which is: ``` {name : “sravan lingam”} ``` ### **Case 1** Let us assume we have only one flow. Let's follow the previous rules. ![Single flow with debugger showing error HTTP:CONNECTIVITY while On Error Continue handles HTTP:NOT_FOUND](../../assets/blog/3-simple-rules-5-examples-to-understand-the-error-handling-in-mule-4-4.png) **Rule1:** There’s something in the Error Handler, but the errorType from the error does not match the errorType from the Error Handling. So it doesn’t matter whether it’s On-Error Continue or On-Error Propagate. When the errorType is not matched, it will use the default error handler by Mule. ``` Actual errorType = HTTP:CONNECTIVITY Handled errorType = HTTP:NOT_FOUND ``` Since none of them match and this is the main flow and it’s not called by any other flow, it uses Mule’s default error-handling. It will print the payload from before the HTTP Request component with a status code 500. Final Response: ``` Response Body: {name : “sravan lingam” } Response status code: 500 ``` ### **Case 2** Let us assume we have only one flow. Let's follow the previous rules. ![Single flow whose On Error Continue handles HTTP:CONNECTIVITY, matching the request error](../../assets/blog/3-simple-rules-5-examples-to-understand-the-error-handling-in-mule-4-5.png) **Rule1:** There’s something in the Error Handler and errorType matches with the handled errorType. ``` Actual errorType = HTTP:CONNECTIVITY Handled errorType = HTTP:CONNECTIVITY ``` Since both match and this is the main flow and it’s not called by any other flow, we can continue with the other two rules. **Rule 2:** The flow is using On-Error Continue. It executes all the components in this block, in this case, we are setting the payload to “Error Handled in Main flow” with a Transform Message component. **Rule 3:** According to points 2 and 3 in Rule 3, since it is On-Error Continue, and the current flow is not called by any other flow, it will execute the processors inside the On-Error Continue and end the process with a 200 status. It will not execute the components after the HTTP Request since this is the main flow. It will print the payload that is set in the Transform Message component from the On-Error Continue. Final Response: ``` Response Body: “Error Handled in Main flow” Response status code: 200 ``` ### **Case 3** If we were to use On-Error Propagate in Case 2 instead of On-Error Continue, every rule is the same but since it’s On-Error Propagate, the error will be raised and sent back to the caller with a status code of 500. Final Response: ``` Response Body: “Error Handled in Main flow” Response status code: 500 ``` ### **Case 4** Let us assume we have 2 flows. Let's follow the previous rules. ![Main flow with On Error Continue and private flow 1 with a Request and On Error Propagate type ANY](../../assets/blog/3-simple-rules-5-examples-to-understand-the-error-handling-in-mule-4-6.png) **Rule1:** There’s something in the Error Handler and the errorType matches with the handled errorType. ``` Actual errorType = HTTP:CONNECTIVITY Handled errorType = ANY ``` > [!NOTE] > ANY can handle any type of error. **Rule 2:** The flow that has the error (private flow 1 in this case) is using On-Error Propagate. It executes all the components in this block, in this case, we are setting the payload to “Error Handled in Private Flow 1” with a Transform message component. **Rule 3:** According to Rule 3, since it is On-Error Propagate, and the current flow is called by the main flow, it will execute the processors inside the On-Error Propagate and then raise the error back to the calling flow, which is the main flow. Now the ball is in the Main flow court. ![Both flows showing the error propagated from private flow 1 back to the main flow's On Error Continue](../../assets/blog/3-simple-rules-5-examples-to-understand-the-error-handling-in-mule-4-7.png) This is how the error is raised back to the Main flow. Repeat the rules: **Rule 1:** The error matches the errorType from the Error Handler. **Rule 2:** On-Error Continue executes the Transform Message component with payload “Error Handled in Main flow”. It will not go further to the other components after the flow-ref. Since it is On-Error Continue, and no other flow is calling the current flow, it will display the latest payload with a 200 status. Final Response: ``` Response Body: “Error Handled in Main flow” Response status code: 200 ``` ### **Case 5** Same as Case 4 except that On-Error Continue is used in private flow 1 (the second flow). ![Main flow with On Error Continue and private flow 1 also using On Error Continue type ANY](../../assets/blog/3-simple-rules-5-examples-to-understand-the-error-handling-in-mule-4-8.png) **Rule 1:** There’s something in the Error Handler and the errorType matches with the handled errorType. ``` Actual errorType = HTTP:CONNECTIVITY Handled errorType = ANY ``` > [!NOTE] > ANY can handle any type of error. **Rule 2:** The flow that has the error (private flow 1 in this case) is using On-Error Continue. It executes all the components in this block, in this case, we are setting the payload to “Error Handled in Private Flow 1” with a Transform message component. **Rule 3:** According to Rule 3, since it is On-Error Continue, and the current flow is called by the main flow, it will execute the processors inside the On-Error Continue, doesn’t raise the error and goes back to the calling flow, which is the main flow. It executes all the components after flow-ref and the final payload will be displayed with a 200 status code. Final Response: ``` Response Body: “Success in Main flow” Response status code: 200 ``` Simple! If you follow this set of rules, then you can easily understand the way that On-Error Continue and On-Error Propagate work! Happy Learning! Sravan Lingam --- ## How to run locally the DataWeave Playground Docker Image Source: https://prostdev.com/post/how-to-run-locally-the-dataweave-playground-docker-image | Published: Aug 11, 2020 | Category: Tutorials You may already know about the DataWeave (DW) Playground that can be used for both versions of DW: 1.0 and 2.0. DataWeave version 1.0 is used in Mule 3, and DataWeave version 2.0 is used in Mule 4. If this is your first time hearing about the DW Playground, check out this post to see why you need to start using it now to test your DataWeave scripts or transformations. I will explain how to get this Docker Image running on your laptop (even if you don’t know how to use Docker). I’ll also show you how to install Docker Desktop and what exact commands you need to run to have it up and ready to use. ## What is the DW Playground, and why is it useful? DataWeave is MuleSoft’s scripting language that is mainly used to transform the data in your integration apps. To be able to use this, you need to download, install and open [Anypoint Studio](https://www.mulesoft.com/platform/studio) (Mule’s [IDE](https://www.prostdev.com/post/5-reasons-why-you-need-an-ide)). Then you need to create a new Mule Project, create a new flow, and finally add the Transform Message component. There are times when you already have a project, and you just want to test out a function to make sure that it works correctly. You can either create a new flow, or create a new project, and then add a new Transform Message component in order to test it out. After you test your code, you normally just delete whatever you created, right? Well, what if there was a way to use a browser, like Google Chrome, and already have your Transform Message component right there to test out the script you want directly? There *is* a way! It’s the DW Playground! Yes, you read that correctly. Let me show you a screenshot of the DW Playground for the 2.0 version. ![DataWeave Playground 2.0 open in Chrome at localhost:9999, outputting "Hello world!"](../../assets/blog/how-to-run-locally-the-dataweave-playground-docker-image-1.png) As you can see, I’m accessing it through Google Chrome, and it’s running on my local computer. There’s no need to open Anypoint Studio to create transformations to practice my DataWeave skills. Talk about convenience! Everything I create using the DW Playground is temporary, so there’s no need to delete any project or files after I finish using it. This Playground is just a Docker Image that can be downloaded to your computer. You can run it with various commands and have it ready to use whenever you want. ## How to install Docker Desktop I’m not going to explain the details of Docker, but I’ll leave some resources at the end of the post to learn more about it if you want to. Just know that previous knowledge in Docker isn’t needed in order to get the Playground running. If you already have Docker Desktop installed, you can skip this part. **1. Download and Install the Docker Desktop app** You can go to [this link](https://www.docker.com/products/docker-desktop) and select your operating system (Windows / Mac) in order to download the corresponding installer. Just follow the instructions to install as you would with any other regular application. **2. Run Docker** Double click on the installed application to open it and make sure that Docker is running in the background. To verify this, you can check your taskbar and look for the Docker icon. ![macOS menu bar with the Docker whale icon highlighted, indicating Docker is running](../../assets/blog/how-to-run-locally-the-dataweave-playground-docker-image-2.png) **3. Open the Docker Desktop application** Once you double click on it, you will see the main screen of the application. Notice that you can see whether Docker is running, and you can also sign in to your DockerHub account. ![Docker Desktop main screen showing "No containers running" with a Sign in button](../../assets/blog/how-to-run-locally-the-dataweave-playground-docker-image-3.png) ## How to run the DataWeave Playground Docker Image Now, let’s get started with the fun part: how to get the DW Playground up and running on our local computer. > [!NOTE] > DockerHub is where all the images are stored for you to download or pull to your local computer. You can create an account if you want to save your favorite images there, but it’s unnecessary. **1. Get the Docker Image from DockerHub** Go to [this link](https://hub.docker.com/r/machaval/dw-playground/tags) so you can see the different versions of the Playground and select the version you want. For this demo, I’ll choose the latest one, which is 2.3.1. On the right side of the Docker Image, you can see there’s a command ready to copy. Feel free to copy this by clicking on the copy button, but don’t run it just yet. ![DockerHub machaval/dw-playground Tags page with the docker pull command for tag 2.3.1 highlighted](../../assets/blog/how-to-run-locally-the-dataweave-playground-docker-image-4.png) **2. Docker Run command** This is what you get when you copy the previous “docker pull” command: ```bash docker pull machaval/dw-playground:2.3.1-SNAPSHOT ``` You just need the last part (the docker image and tag) to paste it into the next docker run command. In this case, machaval/dw-playground:2.3.1-SNAPSHOT. We’ll run the container with this command: ```bash docker run -d --name DWPlayground2 -p 9999:8080 machaval/dw-playground:2.3.1-SNAPSHOT ``` Here is a brief explanation of what you’re running with it: ![Table explaining each part of the docker run command: -d, --name, -p, and the docker image tag](../../assets/blog/how-to-run-locally-the-dataweave-playground-docker-image-5.png) Once the command runs, you will see something like this in your terminal or console output: ![Terminal output of the docker run command pulling the image layers and finishing with the container ID](../../assets/blog/how-to-run-locally-the-dataweave-playground-docker-image-6.png) **3. Verify the container is running in the Docker Desktop app** Once the previous command is done with its execution, you can go to your Docker Desktop application, and you will see the new Docker Container running. ![Docker Desktop showing the DWPlayground2 container running on port 9999](../../assets/blog/how-to-run-locally-the-dataweave-playground-docker-image-7.png) **4. Open the DW Playground** Now you’re ready to open the Playground from your browser! Simply click on the “Open in browser” button from the Docker Desktop app, and this will automatically open the Playground for you. ![Docker Desktop container row with the "Open in browser" action button highlighted](../../assets/blog/how-to-run-locally-the-dataweave-playground-docker-image-8.png) Alternatively, you can manually open your favorite browser and go to this address: ```bash http://localhost:9999/ ``` Note that the port should be replaced with the port you chose in case you changed the 9999 port from the original command. ![DataWeave Playground 2.0 open in Chrome at localhost:9999, outputting "Hello world!"](../../assets/blog/how-to-run-locally-the-dataweave-playground-docker-image-1.png) You're all set! To run the Playground for any other version, you can simply repeat the previous steps. But instead of using machaval/dw-playground:2.3.1-SNAPSHOT you would have to copy the docker image of the version of your preference. For example, to run the version 1.1.8, you could run the following command: ```bash docker run -d --name DWPlayground1 -p 9998:8080 machaval/dw-playground:1.1.8-SNAPSHOT ``` > [!NOTE] > if you intend to run both containers at the same time, you should change the ports to use different ones, so they don’t collide in your local computer. I personally run the DW Playground version 1 on port 9998 and version 2 on port 9999 even when I don’t have them both running at the same time. Here is my Docker Desktop app: ![Docker Desktop with two containers: DWPlayground2 on port 9999 and DWPlayground1 on port 9998](../../assets/blog/how-to-run-locally-the-dataweave-playground-docker-image-9.png) ## How to remove or stop the Docker Containers Maybe you don’t want to use the Playground anymore, or you just want to clear the containers from your app. Luckily for us, the Docker Desktop application provides a button to stop and remove the containers. Take a look at the screenshots below to be able to manipulate your containers. **Stop a running container** ![Docker Desktop with the Stop button highlighted on the running DWPlayground1 container](../../assets/blog/how-to-run-locally-the-dataweave-playground-docker-image-10.png) **Start a container** ![Docker Desktop with the Start button highlighted on the exited DWPlayground1 container](../../assets/blog/how-to-run-locally-the-dataweave-playground-docker-image-11.png) **Delete an existing container** ![Docker Desktop with the Delete button highlighted on the exited DWPlayground1 container](../../assets/blog/how-to-run-locally-the-dataweave-playground-docker-image-12.png) ## Recap In summary, this is what you need to do to run the DataWeave Playground Docker Image: - Download, install and open the Docker Desktop application to get Docker up and running. - Go to DockerHub to find the version of the Playground that you want to download: [https://hub.docker.com/r/machaval/dw-playground/tags](https://hub.docker.com/r/machaval/dw-playground/tags) - Run the docker container with the following command: ```bash docker run -d --name DWPlayground2 -p 9999:8080 machaval/dw-playground:2.3.1-SNAPSHOT ``` *Remember to replace the name (**DWPlayground2**), the port (**9999**), and the DW Playground version (**2.3.1-SNAPSHOT**) with your own settings.* - Open the local container by clicking on the “Open in browser” button from the Docker Desktop app or manually access it through the browser with ```bash http://localhost:9999/ ``` - Stop the running container right from the Docker Desktop app by clicking on the “Stop” button. You can start it again with the “Start” button; there’s no need to re-run the command. I hope this helped you to get started with the DW Playground. Let me know if you had any issues when running these commands or with any of the steps mentioned. *Prost!* -Alex ## More links - [Get Started with Docker](https://www.docker.com/get-started) - [Docker 101 Tutorial](https://www.docker.com/101-tutorial) - [Pluralsight training: Getting Started with Docker](https://www.pluralsight.com/courses/docker-getting-started) - [Docker Cheat Sheet](https://www.docker.com/sites/default/files/d8/2019-09/docker-cheat-sheet.pdf) --- ## Parallel For-Each Scope (Mule 4) Source: https://prostdev.com/post/parallel-for-each-scope-mule-4 | Published: Aug 6, 2020 | Category: Guides Processing a single element of a collection is a very common scenario and for that, we have the For-each scope and Batch processing in MuleSoft. In this article, we will talk about a new scope that is called the Parallel For-each scope. ## Parallel For-Each Scope Like the For-each scope, the Parallel For-each scope also splits the collection of messages into elements. But, unlike the For-each scope, it processes each element simultaneously in separate routes and the result is the collection of all messages aggregated in the same sequence they were before the split. ![Anypoint Studio flow with a Parallel For Each scope wrapping a Try with Is-number and Transform Message](../../assets/blog/parallel-for-each-scope-mule-4-1.png) Looking at the above picture you might be wondering, where are the separate routes? And how are these routes created? The number of routes is equal to the size of the collection and, unlike the scatter-gather, we cannot see these separate routes visually in the canvas. **For example:** The collection configured in the Parallel For-each scope is *[1,2,3,4].* The number of routes Mule runtime will create is 4. ## How does the Parallel For-each work? ![**Each arrow is a separate route inside the Parallel For-each scope and M is the Mule message.](../../assets/blog/parallel-for-each-scope-mule-4-2.png) As shown in the diagram, the collection gets split inside the Parallel For-each scope and each route is processed simultaneously. The output is the collection of Mule messages. > [!NOTE] > The processing of one element is invisible to other elements and creation or modification to an existing variable inside the scope is not visible outside the scope. ![JSON output M1 showing one Mule message with payload number 1 and empty attribute arrays](../../assets/blog/parallel-for-each-scope-mule-4-3.png) ## Configuration The configuration details are well documented in the MuleSoft documentation: [https://docs.mulesoft.com/mule-runtime/4.3/parallel-foreach-scope#configuration](https://docs.mulesoft.com/mule-runtime/4.3/parallel-foreach-scope#configuration) ## Error Handling All routes continue processing even if there is an exception in one of the routes and then runtime executes the code in the error handler. > [!NOTE] > The flow control will not go to the processor placed next to the Parallel For-each scope. ![Diagram comparing success and error scenarios, where an error routes to the Error Handler](../../assets/blog/parallel-for-each-scope-mule-4-4.png) Sometimes we may have a scenario in which we want the flow control to go to the next processor even in case of errors. This functionality can be achieved by using the Try scope and On-Error Continue processor. ![Parallel For Each scope using a Try with an On Error Continue handler so flow reaches the next processor](../../assets/blog/parallel-for-each-scope-mule-4-5.png) ## Conclusion If you want to execute elements of a collection simultaneously without using a batch processing, the Parallel For-each scope is another out of the box solution available in Mule 4. ## References [https://docs.mulesoft.com/mule-runtime/4.3/parallel-foreach-scope](https://docs.mulesoft.com/mule-runtime/4.3/parallel-foreach-scope) --- ## Reviewing Sorting Algorithms: Insertion Sort Source: https://prostdev.com/post/reviewing-sorting-algorithms-insertion-sort | Published: Aug 4, 2020 | Category: Guides Let’s sort it out! In a series of posts, we will be discussing some of the sorting algorithms listed in the below order: - [Bubble Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-bubble-sort) - [Selection Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-selection-sort) - **Insertion Sort** - [Merge Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-merge-sort) - Quick Sort - Heap Sort In this post, we will explore the next in a series of sorting algorithms, the Insertion Sort. If you are still wondering how we landed here with a bunch of sorting algorithms, please go through the previous posts on [Bubble Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-bubble-sort) and [Selection Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-selection-sort). We are yet to discover the fastest and most efficient algorithm to sort the long list of input elements. In this post, the question remains, is Insertion Sort "The One"? We don’t have to wait for the "Oracle" to confirm. Let’s find out. ## Why is it called Insertion Sort? Insertion Sort is one of the simplest and easiest sorting algorithms to understand. By the way, when’s the last time you sorted a hand of playing cards? If it’s today, congratulations! You have understood insertion sort even before we started. Brownie points to you!! If it’s been a while, you should try forming the thinking cloud of sorting the playing cards. You might have not realized it; you were implementing a version of the Insertion Sort algorithm to quickly sort the cards at hand. Well, if you are one of those, who never sorted playing cards, there isn’t a better time than “now” to expedite your understanding of the Insertion Sort. Remember, in our pilot post of [Bubble Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-bubble-sort), we discussed sorting as an act of arranging the elements systematically, based on a predetermined order or certain rules. We have observed from our previously discussed sorting techniques that a comparison of elements is the fundamental idea to arrive at a sorted list. In the case of the Insertion Sort, the basic principle is of inserting an unsorted element at a particular sorted position. The name “Insertion Sort” is derived based upon the concept, where an element has to find its rightful place and has to be inserted in there. If you are confused, let’s go through the below listed instructions to get some clarity. Breaking down the instructions: - For a given unsorted array [10,4,8,6,1], the iteration will be performed for every array element, from left to right. - Two subsets or sub-lists are maintained, the left side is a sorted subset and the right side is an unsorted subset. The first element in the array [10,4,8,6,1] will remain in place and is assumed to be sorted, as there are no further elements on the left side to compare. - Now we have two sections, one sorted section [10], and other unsorted sections [4,8,6,1]. Insertion Sort will iterate through each element of the unsorted list and compare it with the sorted subset, to shift the largest number to the right and insert the smallest element in its current rightful sorted position. - The first element in the unsorted subset (i.e., 4 in our example) is compared against each element from the sorted subset (i.e., 10 in this case) for its rightful sorted position. - If a number in a sorted list is found to be greater than the unsorted list number (i.e., 4), the number is shifted one position right. Meaning, the smaller number is inserted into the sorted subset. Result array would be [4,10,8,6,1]. - The sorting will iterate to the next number (i.e., 8) to compare against a new sorted subset [4,10]. The above steps will be repeated until a valid sorted position is obtained for the same number. - In the next iteration the sorted subset is [4,8,10], the process is repeated for the next number in an unsorted subset [6,1]. We observe that the unsorted subset shrinks with every iteration. Let’s play with our numbers: **Problem:** Sort the given array [10, 4, 8, 6, 1]. As discussed above, we will divide the array into two subsets of sorted and unsorted lists to demonstrate subsets. Iteration starting with the first index [0] as below. ![Insertion sort walkthrough of 10, 4, 8, 6, 1 showing iterations 0 to 2 with sorted and unsorted subsets](../../assets/blog/reviewing-sorting-algorithms-insertion-sort-1.png) ![Insertion sort iterations 3 and 4 shifting elements to reach the final sorted array 1, 4, 6, 8, 10](../../assets/blog/reviewing-sorting-algorithms-insertion-sort-2.png) For more insight into Step 4, below is a simple demonstration. **Observations based on Iterations:** - Well, the obvious observation here is, in each iteration, we compared the first unsorted item against the sorted item to its left to figure out if it’s in the right sorted position. - Iteration is performed on every element and it is from left to right, growing the sorted array. Although the first element is unsorted, it becomes the sorted element in the first iteration. - If the current unsorted array element is smaller than the sorted array element, the current element is shifted to one place higher than its current position. - The array size in our example is n = 5 and the total iterations performed will be (n-1). - The total number of comparisons required to sort: n = 5 elements is *(n-1) + (n-2) + (n-3) + (n-4).* You might recognize the similar pattern from Selection Sort. - In case of a completely sorted list, the Insertion Sort requires the same number of comparisons as that of an unsorted list. However, there will be no shifting or insertion of elements performed. - If there are duplicate elements in the sorting array, the Insertion Sort won’t move one element in front of the other, instead, the key value will be maintained in order. This characteristic makes the Insertion Sort a stable sorting algorithm. You may ask what is considered as stability in the sorting algorithm? ## Defining Stability ![Fig 1.1 Sorting Stability in an algorithm.](../../assets/blog/reviewing-sorting-algorithms-insertion-sort-3.png) ![Fig 1.2 Unstable sorting algorithm.](../../assets/blog/reviewing-sorting-algorithms-insertion-sort-4.png) ## Insertion Vs Selection Vs Bubble Sort ![Comparison table contrasting Insertion, Selection, and Bubble sort behavior](../../assets/blog/reviewing-sorting-algorithms-insertion-sort-5.png) **Time and space complexity** are listed in the below table: ![Table of time and space complexity for Insertion, Selection, and Bubble sort by best, average, and worst case](../../assets/blog/reviewing-sorting-algorithms-insertion-sort-6.png) ## Code Implementation: ```python # Python code for Insertion Sort # Additional Logs are used for demonstration purposes. def InsertionSort(arr): # Get the length of the array arrlen = len(arr) print(' Input Array : {} >> Length : {} '.format(arr,arrlen)) # iterate through the array elements starting from index[1], as we have to compare left index [0] for sorting for i in range(1,arrlen): print ('-----------------------------------------------------') print( 'Current Unsorted array : {} '.format(arr)) print ('At Index: {} >> unsortedValue value {} '.format(i,arr[i])) # Set up j value as a copy of i value j = i # set up the unsorted value to be compared to the left array of sorted values unsortedValue = arr[j] print ('Current unsortedValue array value : {}, value to the left : {}'.format(unsortedValue,arr[j-1])) # Loop from the j (copy of i) element to check for j-1 (left) index value to find the right sorting positon while j > 0 and unsortedValue < arr[j-1]: print ('As {} < {}, Move arr[j-1] value {} to the right at index {} '.format(unsortedValue,arr[j-1],arr[j-1],j)) # As unsortedValue is less then left index value, Move the left index value to right side arr[j] = arr[j-1] print ('At index {} - New value is {} '.format(j,arr[j])) # to check all the left index values, until the list is exhuasted. j = j-1 print (' >> Comparing Left of index: {}, Value {} with unsortedValue: {}'.format(j,arr[j-1],unsortedValue)) # ## ### ## ## # # ### ### # ### ## ## ## # ### # ## ### ## # ## ## # ### # ## ### ## # # The below If condition is not required >> It's added only for demonstration purpose #-------------------------------------------------------------------------------------- if unsortedValue < arr[j-1] and j > 0: print(' >> ({} < {}) => True >> Continue Checking left index values'.format(unsortedValue,arr[j-1])) else: print(' >> ({} < {}) => False and j : {} >> unsortedValue is at sorted position '.format(unsortedValue,arr[j-1],j)) # --------------------------------------------------------------------------------------- print ('At index {} >> Insert unsortedValue value {} '.format(j,unsortedValue)) # Insert unsortedValue to it's rightful position. arr[j]= unsortedValue print ('New array formation : {} '.format(arr)) return arr # Test Long unsorted list #arrlist = [3, 56, 2, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] # Test short unsorted list arrlist = [10,4,8,6,1] #Test sorted list #arrlist = [1,2,3,4,5] #Test n= 100 unsorted list #arrlist = [52,68,84,73,3,81,91,58,1,19,8,78,32,94,4,98,23,6,21,20,43,69,31,87,33,62,36,41,70,51,48,38,72,24,47,57,13,7,64,49,9,16,14,99,90,92,35,25,37,89,50,85,76,67,88,131,39,56,74,42,77,26,93,97,80,82,96,60,53,46,18,65,30,27,2,83,66,29,61,10,55,22,11,75,95,45,59,86,12,15,17,79,40,34,5,71,44,63,28,54] print(InsertionSort(arrlist)) ``` **Log Output:** Let’s check the log output to understand the case of a fully sorted array and unsorted array. ```text #------------------------------------------------------------------- # Test Case 1: Short unsorted list : arrlist = [10,4,8,6,1] #------------------------------------------------------------------- Input Array : [10, 4, 8, 6, 1] >> Length : 5 ----------------------------------------------------- Current Unsorted array : [10, 4, 8, 6, 1] At Index: 1 >> unsortedValue value 4 Current unsortedValue array value : 4, value to the left : 10 As 4 < 10, Move arr[j-1] value 10 to the right at index 1 At index 1 - New value is 10 >> Comparing Left of index: 0, Value 1 with unsortedValue: 4 >> (4 < 1) => False and j : 0 >> unsortedValue is at sorted position At index 0 >> Insert unsortedValue value 4 New array formation : [4, 10, 8, 6, 1] ----------------------------------------------------- Current Unsorted array : [4, 10, 8, 6, 1] At Index: 2 >> unsortedValue value 8 Current unsortedValue array value : 8, value to the left : 10 As 8 < 10, Move arr[j-1] value 10 to the right at index 2 At index 2 - New value is 10 >> Comparing Left of index: 1, Value 4 with unsortedValue: 8 >> (8 < 4) => False and j : 1 >> unsortedValue is at sorted position At index 1 >> Insert unsortedValue value 8 New array formation : [4, 8, 10, 6, 1] ----------------------------------------------------- Current Unsorted array : [4, 8, 10, 6, 1] At Index: 3 >> unsortedValue value 6 Current unsortedValue array value : 6, value to the left : 10 As 6 < 10, Move arr[j-1] value 10 to the right at index 3 At index 3 - New value is 10 >> Comparing Left of index: 2, Value 8 with unsortedValue: 6 >> (6 < 8) => True >> Continue Checking left index values At index 2 >> Insert unsortedValue value 6 As 6 < 8, Move arr[j-1] value 8 to the right at index 2 At index 2 - New value is 8 >> Comparing Left of index: 1, Value 4 with unsortedValue: 6 >> (6 < 4) => False and j : 1 >> unsortedValue is at sorted position At index 1 >> Insert unsortedValue value 6 New array formation : [4, 6, 8, 10, 1] ----------------------------------------------------- Current Unsorted array : [4, 6, 8, 10, 1] At Index: 4 >> unsortedValue value 1 Current unsortedValue array value : 1, value to the left : 10 As 1 < 10, Move arr[j-1] value 10 to the right at index 4 At index 4 - New value is 10 >> Comparing Left of index: 3, Value 8 with unsortedValue: 1 >> (1 < 8) => True >> Continue Checking left index values At index 3 >> Insert unsortedValue value 1 As 1 < 8, Move arr[j-1] value 8 to the right at index 3 At index 3 - New value is 8 >> Comparing Left of index: 2, Value 6 with unsortedValue: 1 >> (1 < 6) => True >> Continue Checking left index values At index 2 >> Insert unsortedValue value 1 As 1 < 6, Move arr[j-1] value 6 to the right at index 2 At index 2 - New value is 6 >> Comparing Left of index: 1, Value 4 with unsortedValue: 1 >> (1 < 4) => True >> Continue Checking left index values At index 1 >> Insert unsortedValue value 1 As 1 < 4, Move arr[j-1] value 4 to the right at index 1 At index 1 - New value is 4 >> Comparing Left of index: 0, Value 10 with unsortedValue: 1 >> (1 < 10) => False and j : 0 >> unsortedValue is at sorted position At index 0 >> Insert unsortedValue value 1 New array formation : [1, 4, 6, 8, 10] [1, 4, 6, 8, 10] #------------------------------------------------------------------------------------------------ # Test Case 2: Long unsorted list : arrlist = [3, 56, 2, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] #------------------------------------------------------------------------------------------------ Input Array : [3, 56, 2, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] >> Length : 13 ----------------------------------------------------- Current Unsorted array : [3, 56, 2, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] At Index: 1 >> unsortedValue value 56 Current unsortedValue array value : 56, value to the left : 3 New array formation : [3, 56, 2, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] ----------------------------------------------------- Current Unsorted array : [3, 56, 2, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] At Index: 2 >> unsortedValue value 2 Current unsortedValue array value : 2, value to the left : 56 As 2 < 56, Move arr[j-1] value 56 to the right at index 2 At index 2 - New value is 56 >> Comparing Left of index: 1, Value 3 with unsortedValue: 2 >> (2 < 3) => True >> Continue Checking left index values At index 1 >> Insert unsortedValue value 2 As 2 < 3, Move arr[j-1] value 3 to the right at index 1 At index 1 - New value is 3 >> Comparing Left of index: 0, Value 45 with unsortedValue: 2 >> (2 < 45) => False and j : 0 >> unsortedValue is at sorted position At index 0 >> Insert unsortedValue value 2 New array formation : [2, 3, 56, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] ----------------------------------------------------- Current Unsorted array : [2, 3, 56, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] At Index: 3 >> unsortedValue value 58 Current unsortedValue array value : 58, value to the left : 56 New array formation : [2, 3, 56, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] ----------------------------------------------------- Current Unsorted array : [2, 3, 56, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] At Index: 4 >> unsortedValue value 79 Current unsortedValue array value : 79, value to the left : 58 New array formation : [2, 3, 56, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] ----------------------------------------------------- Current Unsorted array : [2, 3, 56, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] At Index: 5 >> unsortedValue value 59 Current unsortedValue array value : 59, value to the left : 79 As 59 < 79, Move arr[j-1] value 79 to the right at index 5 At index 5 - New value is 79 >> Comparing Left of index: 4, Value 58 with unsortedValue: 59 >> (59 < 58) => False and j : 4 >> unsortedValue is at sorted postion At index 4 >> Insert unsortedValue value 59 New array formation : [2, 3, 56, 58, 59, 79, 34, 23, 4, 78, 8, 123, 45] ----------------------------------------------------- Current Unsorted array : [2, 3, 56, 58, 59, 79, 34, 23, 4, 78, 8, 123, 45] At Index: 6 >> unsortedValue value 34 Current unsortedValue array value : 34, value to the left : 79 As 34 < 79, Move arr[j-1] value 79 to the right at index 6 At index 6 - New value is 79 >> Comparing Left of index: 5, Value 59 with unsortedValue: 34 >> (34 < 59) => True >> Continue Checking left index values At index 5 >> Insert unsortedValue value 34 As 34 < 59, Move arr[j-1] value 59 to the right at index 5 At index 5 - New value is 59 >> Comparing Left of index: 4, Value 58 with unsortedValue: 34 >> (34 < 58) => True >> Continue Checking left index values At index 4 >> Insert unsortedValue value 34 As 34 < 58, Move arr[j-1] value 58 to the right at index 4 At index 4 - New value is 58 >> Comparing Left of index: 3, Value 56 with unsortedValue: 34 >> (34 < 56) => True >> Continue Checking left index values At index 3 >> Insert unsortedValue value 34 As 34 < 56, Move arr[j-1] value 56 to the right at index 3 At index 3 - New value is 56 >> Comparing Left of index: 2, Value 3 with unsortedValue: 34 >> (34 < 3) => False and j : 2 >> unsortedValue is at sorted position At index 2 >> Insert unsortedValue value 34 New array formation : [2, 3, 34, 56, 58, 59, 79, 23, 4, 78, 8, 123, 45] ----------------------------------------------------- Current Unsorted array : [2, 3, 34, 56, 58, 59, 79, 23, 4, 78, 8, 123, 45] At Index: 7 >> unsortedValue value 23 Current unsortedValue array value : 23, value to the left : 79 As 23 < 79, Move arr[j-1] value 79 to the right at index 7 At index 7 - New value is 79 >> Comparing Left of index: 6, Value 59 with unsortedValue: 23 >> (23 < 59) => True >> Continue Checking left index values At index 6 >> Insert unsortedValue value 23 As 23 < 59, Move arr[j-1] value 59 to the right at index 6 At index 6 - New value is 59 >> Comparing Left of index: 5, Value 58 with unsortedValue: 23 >> (23 < 58) => True >> Continue Checking left index values At index 5 >> Insert unsortedValue value 23 As 23 < 58, Move arr[j-1] value 58 to the right at index 5 At index 5 - New value is 58 >> Comparing Left of index: 4, Value 56 with unsortedValue: 23 >> (23 < 56) => True >> Continue Checking left index values At index 4 >> Insert unsortedValue value 23 As 23 < 56, Move arr[j-1] value 56 to the right at index 4 At index 4 - New value is 56 >> Comparing Left of index: 3, Value 34 with unsortedValue: 23 >> (23 < 34) => True >> Continue Checking left index values At index 3 >> Insert unsortedValue value 23 As 23 < 34, Move arr[j-1] value 34 to the right at index 3 At index 3 - New value is 34 >> Comparing Left of index: 2, Value 3 with unsortedValue: 23 >> (23 < 3) => False and j : 2 >> unsortedValue is at sorted position At index 2 >> Insert unsortedValue value 23 New array formation : [2, 3, 23, 34, 56, 58, 59, 79, 4, 78, 8, 123, 45] ----------------------------------------------------- Current Unsorted array : [2, 3, 23, 34, 56, 58, 59, 79, 4, 78, 8, 123, 45] At Index: 8 >> unsortedValue value 4 Current unsortedValue array value : 4, value to the left : 79 As 4 < 79, Move arr[j-1] value 79 to the right at index 8 At index 8 - New value is 79 >> Comparing Left of index: 7, Value 59 with unsortedValue: 4 >> (4 < 59) => True >> Continue Checking left index values At index 7 >> Insert unsortedValue value 4 As 4 < 59, Move arr[j-1] value 59 to the right at index 7 At index 7 - New value is 59 >> Comparing Left of index: 6, Value 58 with unsortedValue: 4 >> (4 < 58) => True >> Continue Checking left index values At index 6 >> Insert unsortedValue value 4 As 4 < 58, Move arr[j-1] value 58 to the right at index 6 At index 6 - New value is 58 >> Comparing Left of index: 5, Value 56 with unsortedValue: 4 >> (4 < 56) => True >> Continue Checking left index values At index 5 >> Insert unsortedValue value 4 As 4 < 56, Move arr[j-1] value 56 to the right at index 5 At index 5 - New value is 56 >> Comparing Left of index: 4, Value 34 with unsortedValue: 4 >> (4 < 34) => True >> Continue Checking left index values At index 4 >> Insert unsortedValue value 4 As 4 < 34, Move arr[j-1] value 34 to the right at index 4 At index 4 - New value is 34 >> Comparing Left of index: 3, Value 23 with unsortedValue: 4 >> (4 < 23) => True >> Continue Checking left index values At index 3 >> Insert unsortedValue value 4 As 4 < 23, Move arr[j-1] value 23 to the right at index 3 At index 3 - New value is 23 >> Comparing Left of index: 2, Value 3 with unsortedValue: 4 >> (4 < 3) => False and j : 2 >> unsortedValue is at sorted position At index 2 >> Insert unsortedValue value 4 New array formation : [2, 3, 4, 23, 34, 56, 58, 59, 79, 78, 8, 123, 45] ----------------------------------------------------- Current Unsorted array : [2, 3, 4, 23, 34, 56, 58, 59, 79, 78, 8, 123, 45] At Index: 9 >> unsortedValue value 78 Current unsortedValue array value : 78, value to the left : 79 As 78 < 79, Move arr[j-1] value 79 to the right at index 9 At index 9 - New value is 79 >> Comparing Left of index: 8, Value 59 with unsortedValue: 78 >> (78 < 59) => False and j : 8 >> unsortedValue is at sorted position At index 8 >> Insert unsortedValue value 78 New array formation : [2, 3, 4, 23, 34, 56, 58, 59, 78, 79, 8, 123, 45] ----------------------------------------------------- Current Unsorted array : [2, 3, 4, 23, 34, 56, 58, 59, 78, 79, 8, 123, 45] At Index: 10 >> unsortedValue value 8 Current unsortedValue array value : 8, value to the left : 79 As 8 < 79, Move arr[j-1] value 79 to the right at index 10 At index 10 - New value is 79 >> Comparing Left of index: 9, Value 78 with unsortedValue: 8 >> (8 < 78) => True >> Continue Checking left index values At index 9 >> Insert unsortedValue value 8 As 8 < 78, Move arr[j-1] value 78 to the right at index 9 At index 9 - New value is 78 >> Comparing Left of index: 8, Value 59 with unsortedValue: 8 >> (8 < 59) => True >> Continue Checking left index values At index 8 >> Insert unsortedValue value 8 As 8 < 59, Move arr[j-1] value 59 to the right at index 8 At index 8 - New value is 59 >> Comparing Left of index: 7, Value 58 with unsortedValue: 8 >> (8 < 58) => True >> Continue Checking left index values At index 7 >> Insert unsortedValue value 8 As 8 < 58, Move arr[j-1] value 58 to the right at index 7 At index 7 - New value is 58 >> Comparing Left of index: 6, Value 56 with unsortedValue: 8 >> (8 < 56) => True >> Continue Checking left index values At index 6 >> Insert unsortedValue value 8 As 8 < 56, Move arr[j-1] value 56 to the right at index 6 At index 6 - New value is 56 >> Comparing Left of index: 5, Value 34 with unsortedValue: 8 >> (8 < 34) => True >> Continue Checking left index values At index 5 >> Insert unsortedValue value 8 As 8 < 34, Move arr[j-1] value 34 to the right at index 5 At index 5 - New value is 34 >> Comparing Left of index: 4, Value 23 with unsortedValue: 8 >> (8 < 23) => True >> Continue Checking left index values At index 4 >> Insert unsortedValue value 8 As 8 < 23, Move arr[j-1] value 23 to the right at index 4 At index 4 - New value is 23 >> Comparing Left of index: 3, Value 4 with unsortedValue: 8 >> (8 < 4) => False and j : 3 >> unsortedValue is at sorted position At index 3 >> Insert unsortedValue value 8 New array formation : [2, 3, 4, 8, 23, 34, 56, 58, 59, 78, 79, 123, 45] ----------------------------------------------------- Current Unsorted array : [2, 3, 4, 8, 23, 34, 56, 58, 59, 78, 79, 123, 45] At Index: 11 >> unsortedValue value 123 Current unsortedValue array value : 123, value to the left : 79 New array formation : [2, 3, 4, 8, 23, 34, 56, 58, 59, 78, 79, 123, 45] ----------------------------------------------------- Current Unsorted array : [2, 3, 4, 8, 23, 34, 56, 58, 59, 78, 79, 123, 45] At Index: 12 >> unsortedValue value 45 Current unsortedValue array value : 45, value to the left : 123 As 45 < 123, Move arr[j-1] value 123 to the right at index 12 At index 12 - New value is 123 >> Comparing Left of index: 11, Value 79 with unsortedValue: 45 >> (45 < 79) => True >> Continue Checking left index values At index 11 >> Insert unsortedValue value 45 As 45 < 79, Move arr[j-1] value 79 to the right at index 11 At index 11 - New value is 79 >> Comparing Left of index: 10, Value 78 with unsortedValue: 45 >> (45 < 78) => True >> Continue Checking left index values At index 10 >> Insert unsortedValue value 45 As 45 < 78, Move arr[j-1] value 78 to the right at index 10 At index 10 - New value is 78 >> Comparing Left of index: 9, Value 59 with unsortedValue: 45 >> (45 < 59) => True >> Continue Checking left index values At index 9 >> Insert unsortedValue value 45 As 45 < 59, Move arr[j-1] value 59 to the right at index 9 At index 9 - New value is 59 >> Comparing Left of index: 8, Value 58 with unsortedValue: 45 >> (45 < 58) => True >> Continue Checking left index values At index 8 >> Insert unsortedValue value 45 As 45 < 58, Move arr[j-1] value 58 to the right at index 8 At index 8 - New value is 58 >> Comparing Left of index: 7, Value 56 with unsortedValue: 45 >> (45 < 56) => True >> Continue Checking left index values At index 7 >> Insert unsortedValue value 45 As 45 < 56, Move arr[j-1] value 56 to the right at index 7 At index 7 - New value is 56 >> Comparing Left of index: 6, Value 34 with unsortedValue: 45 >> (45 < 34) => False and j : 6 >> unsortedValue is at sorted position At index 6 >> Insert unsortedValue value 45 New array formation : [2, 3, 4, 8, 23, 34, 45, 56, 58, 59, 78, 79, 123] [2, 3, 4, 8, 23, 34, 45, 56, 58, 59, 78, 79, 123] #------------------------------------------------------------------- # Test Case 3: sorted list : arrlist = [1,2,3,4,5] #------------------------------------------------------------------- Input Array : [1, 2, 3, 4, 5] >> Length : 5 ----------------------------------------------------- Current Unsorted array : [1, 2, 3, 4, 5] At Index: 1 >> unsortedValue value 2 Current unsortedValue array value : 2, value to the left : 1 New array formation : [1, 2, 3, 4, 5] ----------------------------------------------------- Current Unsorted array : [1, 2, 3, 4, 5] At Index: 2 >> unsortedValue value 3 Current unsortedValue array value : 3, value to the left : 2 New array formation : [1, 2, 3, 4, 5] ----------------------------------------------------- Current Unsorted array : [1, 2, 3, 4, 5] At Index: 3 >> unsortedValue value 4 Current unsortedValue array value : 4, value to the left : 3 New array formation : [1, 2, 3, 4, 5] ----------------------------------------------------- Current Unsorted array : [1, 2, 3, 4, 5] At Index: 4 >> unsortedValue value 5 Current unsortedValue array value : 5, value to the left : 4 New array formation : [1, 2, 3, 4, 5] [1, 2, 3, 4, 5] ``` **Unsorted array:** There are two loops performed, one is the outer loop for array iteration, and another is the inner loop for comparisons. At the start of the sorting, the first element (10) is considered as a sorted subset and the inner loop will compare the first unsorted element (4) against the sorted element. As the unsorted array shrinks on every iteration, the sorted array inner loop keeps growing to shift the elements. The inner loop will run for *(n-1)* elements wherein *‘n’* is the total number of array elements. For instance, for the last iteration in the test case, the array index [4] = 1 element is compared against (n-1) = 4 sorted array elements. If you observe closely, the inner loops iterate from right to left to compare against the sorted array list. The double looping is what makes the Insertion Sort worst-case time complexity О(*n^2*) because if the input doubles the running time will be quadratic. **Sorted array:** In test case 3, the log shows no inner loops, as the values are already sorted. However, the comparisons are performed to check the rightful sorting position. Insertion Sort requires ‘n’ steps to sort an already sorted array of ‘n’ elements, which makes its best-case time complexity a linear function of O(n). ## Final thoughts So next time you sort out cards manually or play bridge, remember it’s an Insertion Sort. It is yet another simple algorithm that is slow due to its internal looping. In all fairness, Insertion Sort has an edge over its peers Bubble and Selection Sort as it has a best-case running time for an already sorted array list. Moreover, it is a stable algorithm with in-place properties and the ability to sort a list as it is received. We have arrived at the most awaited conclusion, the Insertion Sort is not “The One” we were looking out for. Well, our quest to find “The One” continues and looks like we are getting really close. In the next post, we shall discuss “Merge Sort” to continue our pursuit of finding the most efficient sorting algorithm. Stay tuned!! I hope you enjoyed the post. Please subscribe to [ProstDev](https://www.prostdev.com/) for more exciting topics. Hasta luego, amigos! ## Study Resources Below are some resources which discuss Insertion Sort and its Time-Space complexity at length. - [The Insertion Sort - Runestone academy](https://runestone.academy/runestone/books/published/pythonds/SortSearch/TheInsertionSort.html) - [Insertion Sort - Wikipedia](https://en.wikipedia.org/wiki/Insertion_sort) - [Video] [Insertion Sort - CS50 Harvard](https://www.youtube.com/watch?v=O0VbBkUvriI) - [Analysis of Insertion Sort - Khan Academy](https://www.khanacademy.org/computing/computer-science/algorithms/insertion-sort/a/analysis-of-insertion-sort) - [Video] [Insertion Sort - CS310 San Diego University](https://www.youtube.com/watch?v=eTvQIbB-AuE) - [Video] [Insert Sort with Romanian Folk Dance](https://www.youtube.com/watch?v=ROalU379l3U) - [Asymptotic-Notation- Khan Academy](https://www.khanacademy.org/computing/computer-science/algorithms/asymptotic-notation/a/asymptotic-notation) - [Big-O cheat sheet](https://www.bigocheatsheet.com/) - [Insertion Sort Animation](https://www.toptal.com/developers/sorting-algorithms/insertion-sort) - [Stable Sorting Algorithms](https://www.baeldung.com/cs/stable-sorting-algorithms) - [Sorting Algorithms Stability](https://en.wikipedia.org/wiki/Sorting_algorithm#Stability) - [A Comparative Study of Selection Sort and Insertion Sort Algorithms](https://www.irjet.net/archives/V3/i12/IRJET-V3I12115.pdf) Fahriye Gemci FuratResearch Assistant, Department of Computer Engineering, Iskenderun Technical University, Hatay, Turkey --- ## Dell Boomi Integrations: Using the Atom Queue Connector Source: https://prostdev.com/post/dell-boomi-integrations-using-the-atom-queue-connector | Published: Jul 28, 2020 | Category: Tutorials The Atom Queue connector is used to send and receive messages to and from native Boomi Integration Atom message queues. The connector supports both Point-to-Point and Publish/Subscribe messaging. For better understanding of how Atom Queue works, I did a proof of concept. Below are the screenshots of the flow which will give you a better idea of how this component works. ## 1. Creating a message and sending it to an Atom Queue To send a message to another process, first you need to create the process where we’ll send the queue message from (in this case, we create the Send_Queue process). ![Boomi Send_Queue process: Start (No Data), Message, Notify, Atom Queue, then Stop](../../assets/blog/dell-boomi-integrations-using-the-atom-queue-connector-1.png) The “No Data” option should be selected in the “Start” Shape connector, if the process will not receive or retrieve data from any source. In the Message connector, I have entered a “Hello” message (as shown below). This is the message that will be sent to the second process. ![Boomi Message Shape config holding a JSON "Hello" demo message](../../assets/blog/dell-boomi-integrations-using-the-atom-queue-connector-2.png) The message sent from the Message Shape is sent to the Atom Queue connector, which will be sending the message to native Boomi Integration Atom Message Queues. ![Boomi Connector Shape set to Atom Queue connector with the Send action](../../assets/blog/dell-boomi-integrations-using-the-atom-queue-connector-3.png) To configure a connector to send a message to an Atom Message Queue, I need to set up two components: - Atom Queue Connection - Atom Queue Operation The Atom Queue Connection specifies the message queue for an Atom Queue Operation. As shown below, I have configured the Atom Queue Connection with the Queue Name “AtomQueue_Test”. ![New Atom Queue Connection config with Queue set to AtomQueue_Test](../../assets/blog/dell-boomi-integrations-using-the-atom-queue-connector-4.png) The Atom Queue Operation below is used to send (this is the Connection Action) messages to a Message Queue and Batch Size is set to 5 (default value), which means it sets the number of documents to batch in each outgoing message. ![Atom Queue Operation set to the Send action with Batch Size of 5](../../assets/blog/dell-boomi-integrations-using-the-atom-queue-connector-5.png) The Stop Shape indicates the “End and continue” of the flow. ## 2. Reading the message from the Atom Queue After we send the message to the queue, the next step would be to read that message from the second flow. I created the below process (Read_Queue) to read the file that was previously sent to the queue. ![Boomi Read_Queue process: Start (Atom Queue), Notify, then Stop](../../assets/blog/dell-boomi-integrations-using-the-atom-queue-connector-6.png) In the Start Shape connector, “Atom Queue” must be selected as the connector, and the Action must be selected as “Listen”. ![Boomi Start Shape with Atom Queue connector and the Listen action selected](../../assets/blog/dell-boomi-integrations-using-the-atom-queue-connector-7.png) The connection will be the same as the Atom Queue Connection we used in the Send Process, and the operation will be set to Listen (Connector Action). “Listen” is set in event-driven retrieval of messages from a Message Queue. To receive first-in, first-out (FIFO) processing of messages across nodes, set the Maximum Concurrent Executions value to 1. ![Atom Queue Operation set to Listen with Maximum Concurrent Executions of 1](../../assets/blog/dell-boomi-integrations-using-the-atom-queue-connector-8.png) I used a Notify Shape to send a customized message. Here I am just using the Current Data option, which will print the message that was received from the Atom Queue. ![Boomi Notify Shape config set to print the Current Data message](../../assets/blog/dell-boomi-integrations-using-the-atom-queue-connector-9.png) After these two processes are ready, the next step is to create a packaged component for the “Read_Queue” (Listen) process and deploy it. After deploying, the process can be seen in the Atom Management page -> Listeners, which means the process is in Listen mode. ![Atom Management Listeners page showing Read_Queue in listen mode](../../assets/blog/dell-boomi-integrations-using-the-atom-queue-connector-10.png) Next step is to test the process in Test Mode. ![Boomi Test dialog with a Test Atom Cloud selected and a Run Test button](../../assets/blog/dell-boomi-integrations-using-the-atom-queue-connector-11.png) Since the “Read_Queue” is already deployed, I will run the first process (in this case, “Send_Queue”). After the execution is completed without any errors, the message will be sent to the queue, which can be seen from the “Process Reporting” tab, as shown below. ![Boomi Executions report showing a completed Read_Queue process run](../../assets/blog/dell-boomi-integrations-using-the-atom-queue-connector-12.png) Click on the “View Process Logs” icon. The message can be seen from the Notify Shape, which prints the data that is passed from the first process (Send_Queue). ![Boomi process log showing the Notify shape printing the received "Hello" message](../../assets/blog/dell-boomi-integrations-using-the-atom-queue-connector-13.png) ## Errors Common errors when using Atom Queues are: - If message delivery fails, the Dead Letter Queue will be created corresponding to the Destination Queue and the failed messages will be placed in the Dead Letter Queue after 6 retry attempts by the shared queue server of the same Atom Queue component. The messages in the destination Dead Letter Queue can be manually retried from the “Queue Management” tab, via the Platform UI. - If the destination is other than an Atom Queue Server and there is a need to catch the failed messages in case of any failures, the best way to handle this scenario would be to re-queue the failed messages in the catch branch of a try/catch. They can either be placed back on the original queue or they could be placed on some sort of retry queue. --- ## Understanding the "illegal base64 character" error (Java, Groovy and Mule 4 - DW 2.0) Source: https://prostdev.com/post/understanding-the-illegal-base64-character-error-java-groovy-and-mule-4-dw-2-0 | Published: Jul 21, 2020 | Category: Tutorials Are you familiar with the fromBase64 or the toBase64 functions from DataWeave 2.0? What about the getUrlEncoder or the getEncoder functions from Java? Do you know the differences between the “basic” Base 64 encoding and the “URL and Filename safe” Base 64 encoding? Well, you may have guessed it by now, but you’re about to find out the answers to these questions! Or maybe you’re here because you keep getting the “Illegal base64 character” error in DataWeave. Even if you get this error using any other programming language, this post can help you understand why it is happening. ## The Problem I was recently presented with this problem when Maria Isabel Vargas asked a question about base64 decoding in a Slack channel. She was using the [fromBase64](https://docs.mulesoft.com/mule-runtime/4.3/dw-binaries-functions-frombase64) function that’s available in Mule 4 - DataWeave 2.0 inside the [dw::core::Binaries](https://docs.mulesoft.com/mule-runtime/4.3/dw-binaries) module to transform a basic Base 64 string into a binary value. The problem was that the server was returning a Base 64 **URL** Encoded String (e.g. “cHJvc3RkZXY**_**YmxvZw==”) opposed to the **basic** Base 64 string from which the fromBase64 function attempts to transform. As a result, the Transform Message component was returning this error: “Illegal base64 character 5f". This error happens when the string that you are trying to transform contains a character not recognized by the **basic** Base 64 Alphabet (in this case it was an underscore character). Below you can see which characters are accepted. ![ietf.org - RFC 4648](../../assets/blog/understanding-the-illegal-base64-character-error-java-groovy-and-mule-4-dw-2-0-2.png) Since the server was using the Base 64 **URL** encoding, the string that we were trying to decode in DataWeave contained different characters from the ones above because the Base 64 URL has a different alphabet. You can see this alphabet below. ![ietf.org - RFC 4648](../../assets/blog/understanding-the-illegal-base64-character-error-java-groovy-and-mule-4-dw-2-0-3.png) Notice that the characters 62 and 63 differ from the **basic** Base 64 Alphabet and the Base 64 **URL** Alphabet. The first one contains the characters plus (+) and slash (/), while the second one uses the characters minus (-) and underline (_). In other words, if you have an encoded string like "cHJvc3RkZXY_YmxvZw==", you wouldn’t be able to transform it using a **basic** Base64 decoder because it contains characters that are not recognized by its alphabet (the underline character). You can get these two errors when you try to transform a Base 64 **URL** string using the fromBase64 DW function (at least when this post was created): - java.lang.IllegalArgumentException: Illegal base64 character 5f (when containing an underscore) - java.lang.IllegalArgumentException: Illegal base64 character 2d (when containing a minus) ![DataWeave Playground running fromBase64 on a URL-encoded string and throwing an Illegal base64 character error.](../../assets/blog/understanding-the-illegal-base64-character-error-java-groovy-and-mule-4-dw-2-0-4.png) ## The Solution Java contains a function called [getUrlDecoder](https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html#getUrlEncoder--), which is used to decode a string that was encoded using the Base 64 **URL** alphabet. This is exactly the problem that we were trying to solve. But before you start panicking and coding your solution using Java and creating a whole Java class, it's way easier to just use the [Mule 4 Scripting Module](https://docs.mulesoft.com/scripting-module/1.1/). I’ll show you how. This module should already be installed in your Studio. However, if you can’t see it, you can download it from Exchange. Alternatively, you can use the “Add Modules” button from Anypoint Studio, which is located on your Mule Palette. ![Anypoint Studio Mule Palette with the Add Modules and Search in Exchange options.](../../assets/blog/understanding-the-illegal-base64-character-error-java-groovy-and-mule-4-dw-2-0-5.png) After that, you can simply take the Execute component and drag-and-drop it into your flow. For this demonstration, I will be using an HTTP Listener as a trigger, and I will send the encoded string in the body of the request from Postman. Once you have the Execute component, you can select “Groovy” as your Engine and paste the following line of code into your script: ```groovy Base64.getUrlDecoder().decode(payload) ``` You should end up with this: ![Flow with an HTTP Listener and an Execute component whose Groovy code calls Base64.getUrlDecoder().decode(payload).](../../assets/blog/understanding-the-illegal-base64-character-error-java-groovy-and-mule-4-dw-2-0-6.png) As you can see in this Groovy script, we are using the getUrlDecoder function to decode whatever is in our payload. The result will be returned to our Postman application in the body of the response. You need to make sure that the payload that is sent to the Groovy script is a Java String. So, let’s add a Transform Message right after the HTTP Listener and transform the payload into a Java string. Like this: ![Flow with a Transform Message added between the Listener and Execute, casting the payload as String for Java.](../../assets/blog/understanding-the-illegal-base64-character-error-java-groovy-and-mule-4-dw-2-0-7.png) Let’s start this Mule app and test! We send the same string as before (cHJvc3RkZXY_YmxvZw==) from Postman, and we expect to get the decoded message as a response. ![Postman POST to localhost:8081/flow1 returning 200 OK with the decoded "prostdev?blog" response.](../../assets/blog/understanding-the-illegal-base64-character-error-java-groovy-and-mule-4-dw-2-0-8.png) And… it worked! ## Try it yourself Do you want to experiment a bit more with all of this? For sure! Let me give you a quick guide on how to do this. ### 1. Encode the bytes/string into a Basic 64 URL encoded string You will need Java (or you can also do it in Mule 4 using a Groovy script) to get the encoded string first. You don’t need to download and install Java onto your computer to do this; you can search for a free online Java compiler like this one: [TutorialsPoint Java Compiler](https://www.tutorialspoint.com/compile_java_online.php) There, just copy and paste this Java code: ```java import java.util.Base64; public class Base64UrlEncoding { public static void main(String []args) { byte[] bytes = "prostdev?blog".getBytes(); String encodedString = new String(Base64.getUrlEncoder().encode(bytes)); System.out.println(encodedString); } } ``` Next, click on “Execute” in the top left corner. You will see the encoded string on the right side of the website (the last line in the white background). Copy this string and save it somewhere so that you can use it in the next steps. ![Online Java compiler running the Base64UrlEncoding class, printing the encoded string cHJvc3RkZXY_YmxvZw==.](../../assets/blog/understanding-the-illegal-base64-character-error-java-groovy-and-mule-4-dw-2-0-9.png) ### 2. Create your Mule App in Anypoint Studio Follow the steps explained previously in “The Solution”, or paste this code into the XML configuration of your file: ```xml ``` ### 3. Create a Postman request If you don’t have Postman, you can download it for free from here: [Postman Download](https://www.postman.com/downloads/). When you open it, you just have to click on the plus (+) button as you can see here: ![Postman with no collections yet and an arrow pointing at the plus button to create a new request.](../../assets/blog/understanding-the-illegal-base64-character-error-java-groovy-and-mule-4-dw-2-0-10.png) Then, add this in the Request URL section: ``` localhost:8081/flow1 ``` And add the previously copied string inside the Request Body. Don’t forget to select “Raw” and “Text” from the two dropdowns that are over the request body text area. ![Postman request set to localhost:8081/flow1 with the encoded string in a raw Text request body.](../../assets/blog/understanding-the-illegal-base64-character-error-java-groovy-and-mule-4-dw-2-0-11.png) ### 4. Run your Mule App Once it is deployed successfully, you can click on the big blue “Send” button from Postman and see the results which should match the string you encoded in Step 1 (using Java). ![Anypoint Studio console showing the testbase64 app starting and reaching DEPLOYED status.](../../assets/blog/understanding-the-illegal-base64-character-error-java-groovy-and-mule-4-dw-2-0-12.png) ![Postman POST to localhost:8081/flow1 returning 200 OK with the decoded "prostdev?blog" response.](../../assets/blog/understanding-the-illegal-base64-character-error-java-groovy-and-mule-4-dw-2-0-8.png) ## Recap Let’s do a quick recap on what we just learned from this post: - The **basic** Base 64 alphabet can recognize the characters plus (+) and slash (/). - The Base 64 **URL** alphabet can recognize the characters minus (-) and underline (_). - Depending on how a string was encoded (using basic Base 64 or Base 64 URL encoding), it can include any of the characters from the chosen alphabet. - An encoded string with one alphabet may not be decoded using the other alphabet’s decoding system. If the decoder attempts to read an unrecognized character, it’ll send an Illegal base64 character error message. - You **can’t** decode a Base 64 **URL** string with the fromBase64 DataWeave 2.0 function. - You **can** decode a Base 64 **URL** string by using an Execute component along with a Groovy engine and the getUrlDecoder Java function. Had you seen this error before? How did you solve it? *Prost!* -Alex ## References - [Mule docs - fromBase64 function](https://docs.mulesoft.com/mule-runtime/4.3/dw-binaries-functions-frombase64) - [Mule docs - dw::core::Binaries](https://docs.mulesoft.com/mule-runtime/4.3/dw-binaries) - [Mule docs - Scripting Module](https://docs.mulesoft.com/scripting-module/1.1/) - [IETF - RFC4648](https://www.ietf.org/rfc/rfc4648.txt) - [Oracle docs - getUrlEncoder](https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html#getUrlEncoder--) - [TutorialsPoint - Online Java compiler](https://www.tutorialspoint.com/compile_java_online.php) - [Download Postman](https://www.postman.com/downloads/) ### GitHub repository [ProstDev GitHub - testbase64](https://github.com/ProstDev/testbase64) --- ## Custom Modules in Mule 4 - DataWeave 2.0 Source: https://prostdev.com/post/custom-modules-in-mule-4-dataweave-2-0 | Published: Jul 16, 2020 | Category: Tutorials *GitHub repository with the Mule Project can be found at the end of the post.* Some days ago, I started using this great functionality in DW 2.0, and I faced some challenges while creating the code, so I figured I’d write a blog post about my experience. [MuleSoft’s documentation](https://docs.mulesoft.com/mule-runtime/4.3/dataweave-create-module) offers a great explanation of how to create these custom modules. One problem that I couldn’t find how to solve in the documentation was how to correctly load a property from the properties file into the custom module. If you’re not familiar with custom modules, don’t worry, I got your back ;) For this post, I’ll explain what a custom module is, why you would want to use it, some examples on how you can use it, and finally, how to load properties into this module. ## What are Custom Modules in DataWeave 2.0? First of all, let me tell you what a regular Module is in DW 2.0: from [MuleSoft’s documentation](https://docs.mulesoft.com/mule-runtime/4.3/dw-functions), “*DataWeave 2.0 functions are packaged in modules. Functions in the Core (**dw::Core**) module are imported automatically into your DataWeave scripts.*”. Yes, all those functions that you use in your DW scripts, come from the Core module. Functions like [++](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-plusplus), [flatten](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-flatten), [map](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-map), [joinBy](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-joinby), that you may use on your day-to-day, come from Modules, you just don’t need to explicitly import them into your script. Other developers may call them [libraries](https://en.wikipedia.org/wiki/Library_(computing)#:~:text=In%20computer%20science%2C%20a%20library,classes%2C%20values%20or%20type%20specifications.) however in DW 2.0 they are called modules. Mule offers a wide choice of modules that you can use to transform and manipulate your data, like the [Arrays](https://docs.mulesoft.com/mule-runtime/4.3/dw-arrays) module, the [Strings](https://docs.mulesoft.com/mule-runtime/4.3/dw-strings) module, the [Objects](https://docs.mulesoft.com/mule-runtime/4.3/dw-objects) module, even [Binaries](https://docs.mulesoft.com/mule-runtime/4.3/dw-binaries) or [Tree](https://docs.mulesoft.com/mule-runtime/4.3/dw-tree). You don’t need to know how all of these work, just understand the basic meaning of a module, which is: functions created by MuleSoft that can be reused into your DW scripts. Custom Modules are pieces of code that can be stored into an external file, in order to be reused in other parts of the Mule application. These are not created by MuleSoft, but by yourself, or your company. ## Why are they useful? Here is an example to illustrate why Custom Modules are useful: you just created a new function that will join an array of words into a complete sentence. Like this: ![Transform Message joining a words array with joinBy into the sentence "Hello ProstDev :)"](../../assets/blog/custom-modules-in-mule-4-dataweave-2-0-1.png) As you can see, we are using the joinBy function in the code to select which character we want our words to be separated by, in our complete sentence. If we were to change our code from joinBy “ “ to joinBy “-”, our final output would be "Hello-ProstDev-:)" instead. If this piece of code, to create a complete sentence from an array of words, was used in several parts of your application, it would be a good idea to create a function inside a Custom Module to store this code. Then just reference that function from anywhere else in the rest of the app; instead of having to copy and paste this code into each part where you need to use it. Let’s see how we can create this module for our Mule Application: - Create a “modules” folder inside your `src/main/resources` folder, and then create a new “.dwl” file inside the new modules folder, you can name it something like `Custom.dwl` (you can name the folder and the file however you prefer). ![Project tree showing a new Custom.dwl file inside a modules folder under src/main/resources](../../assets/blog/custom-modules-in-mule-4-dataweave-2-0-2.png) - Inside this file, create a new function and paste the code that we had just created to generate the output sentence from our words array (note that you don’t need to add the “output” data type, or the 3 dashes (-) before the code). ![Custom.dwl defining a getSentenceFromWords function that joins words by a space](../../assets/blog/custom-modules-in-mule-4-dataweave-2-0-3.png) - Save everything, and then go back to the Transform Message where we have our DW script. Import the new function from the “Custom” module by writing import getSentenceFromWords from modules::Custom under the output statement. Now you can simply call this function in your code to use the Custom Module and we’ll get the same results as before. ![Transform Message importing getSentenceFromWords from modules::Custom and calling it](../../assets/blog/custom-modules-in-mule-4-dataweave-2-0-4.png) Of course, this is a very simple example, it’s only one line of code. You may think that we generated even more code by doing this than by copying and pasting the code in several scripts. Fair enough. But, imagine what would happen if you suddenly decide to change the functionality? You’d have to go to each of the DW scripts and manually change every single line of code. If you have a custom module setup, you’d only have to change the function from the `Custom.dwl` file, and all the scripts referencing it will automatically have the new functionality. Copy and Pasting means you have to go through and change each one but with this method it's done automatically - for graphic designers out there it's like the embed feature in photoshop. ## Loading properties into a Custom Module Remember that character we used after the joinBy function? What if we wanted that character to be in a properties file, instead of hardcoded (added directly) in our script? Can we do that when using Custom Modules? Yes, we can! I already have my `default-properties.yaml` file created under `src/main/resources`, with one property that I decided to name “joinByChar” ![default-properties.yaml file defining a joinByChar property set to a space](../../assets/blog/custom-modules-in-mule-4-dataweave-2-0-5.png) We can go back to our `Custom.dwl` file and change the script to reference this property instead of hardcoding the character there. You may already know that the properties can be used in your Mule Application by using either the ${ } or the p(‘ ‘) syntax. However, if you try to use any of these from a Custom Module, DataSense will send you back an error (at least for Mule Runtime version 4.3.0 and Anypoint Studio version 7.5.0). I was losing my mind trying to search in the [Mule Docs](https://docs.mulesoft.com/general/) or in the [Mule Forums](https://help.mulesoft.com/s/forum) how I could import a property from my custom module. In the end, one of the [Mule Ambassadors](https://developer.mulesoft.com/dev/ambassadors), Manish Yadav, was able to point me in the right direction. Remember at the beginning of this post, when I explained that all the DataWeave functions come from a module that is created by MuleSoft. Well, when you use this function: p(‘my-property’) to load a property into the script, do you know which module it comes from? It comes from the [Mule module](https://docs.mulesoft.com/mule-runtime/4.3/dw-mule)! We can rewrite our function like this: ![Custom.dwl rewritten to read the separator via Mule::p('joinByChar') instead of hardcoding it](../../assets/blog/custom-modules-in-mule-4-dataweave-2-0-6.png) Aaaaand… Magic! We have our same output, but now we’re using an external property to generate the final sentence. ![Mule flow and Transform Message still output "Hello ProstDev :)" using the space property](../../assets/blog/custom-modules-in-mule-4-dataweave-2-0-7.png) This way we can change our property joinByChar right from the `default-properties.yaml` file and we do not need to touch the code to select a new character. ![Changing joinByChar to a dash makes the output become "Hello-ProstDev-:)" with no code edits](../../assets/blog/custom-modules-in-mule-4-dataweave-2-0-8.png) Feel free to download the Mule project from our github repository and give it a try! Maybe you get it to work with another syntax :) I’d love to keep learning more DataWeave functions. ## Recap Let’s do a quick recap on what we learned with this post: - DataWeave 2.0 functions are packaged in modules. - Modules are created by MuleSoft and you can use their predefined functions in your DW scripts by importing the module, using the import keyword. - Custom Modules are just like regular modules, but they are created by you or your company, as opposed to MuleSoft. - Modules are useful because they let you reuse the same piece of code in different DW scripts. - Custom Modules are stored into an external file and can be imported into your DW script the same way that a regular module is imported: using the import keyword. Special thanks to Manish for this tip and for being so quick to respond to my messages! Stay tuned for more DataWeave tips and tricks. *Prost!* -Alex ## References - [Mule docs - MuleSoft Documentation](https://docs.mulesoft.com/general/) - [Mule docs - Create Custom Modules and Mappings](https://docs.mulesoft.com/mule-runtime/4.3/dataweave-create-module) - [Mule docs - DataWeave Reference](https://docs.mulesoft.com/mule-runtime/4.3/dw-functions) - [Mule docs - joinBy function](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-joinby) - [Mule docs - Mule Module](https://docs.mulesoft.com/mule-runtime/4.3/dw-mule) - [Mule Forum](https://help.mulesoft.com/s/forum) - [Mule Ambassadors](https://developer.mulesoft.com/dev/ambassadors) - [Wikipedia - Library](https://en.wikipedia.org/wiki/Library_(computing)#:~:text=In%20computer%20science%2C%20a%20library,classes%2C%20values%20or%20type%20specifications) ## GitHub repository [ProstDev GitHub - Custom Modules DW 2.0](https://github.com/ProstDev/custom-modules-dw2) --- ## Reviewing Sorting Algorithms: Selection Sort Source: https://prostdev.com/post/reviewing-sorting-algorithms-selection-sort | Published: Jul 14, 2020 | Category: Guides Let's sort it out! In a series of posts, we will be discussing some of the sorting algorithms listed in the below order: - [Bubble Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-bubble-sort) - **Selection Sort** - [Insertion Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-insertion-sort) - [Merge Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-merge-sort) - Quick Sort - Heap Sort In this post, we will explore the next, in a series of sorting algorithms, the selection sort. If you feel nostalgic about sorting algorithms, I don’t blame you and it’s totally fine if you want to skim through the first post of the series on [Bubble Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-bubble-sort). Good read eh! Now you know that bubble sort’s time and space complexity is kind of meh! We crave for a faster and more efficient algorithm to sort the long list of input elements. Is “Selection Sort” the answer? Maybe. ## Why is it called Selection Sort? The sorting algorithm gets its name from the iterations it performs through input arrays or elements. The algorithm selects the current smallest element and swaps it into its sorted place and continues doing it until the list is sorted. It creates two sections of the arrays; one section is unsorted, and another is sorted. In short, we will be selecting one item at a time by its size and moving it to its sorted position. Breaking down the instructions: - Given an unsorted array, set the first index [0] element as the smallest number. - Iterate through the array to find the smallest number. - Now swap the smallest number as Index [0] and the original index [0] value to index of the smallest number. - Continue the above process for the rest of the unsorted array until the last element is reached. I know, it’s like “yeah I get it, but wait I’m lost” and that’s why we have visuals! Yay! We will pick up the same numbers we introduced during the bubble sort as an example. Yes! I’m trying to make you compare the techniques already. Let’s play with the selections: **Problem:** Sort the numbers [10, 4, 8, 6, 1]. Let’s start with the first *iteration* of the unsorted list based on our understanding of the above instruction. ![Step-by-step selection sort iterations sorting the array [10, 4, 8, 6, 1]](../../assets/blog/reviewing-sorting-algorithms-selection-sort-1.png) **Observations based on Iterations:** - In our example, the array size is *n=5* and 4 iterations were performed. Similarly, for *‘n’* numbers *‘n-1’* iterations will be performed. - Only one element gets sorted on each pass. - The number of comparisons needed for first iterations was *(n-1),* as we compared 4 elements to find the smallest number. Likewise, comparisons needed for the second iteration is (n-2), third is (n-3), and so on. - The total number of comparisons required to sort n=5 elements are *(n-1) +(n-2) +(n-3) +(n-4)*. From our example problem length of array is *n= 5,* no. of comparisons is 4+3+2+1 = 10. - Summation of the above stated turns out to be *‘n(n-1)/2’*. So, guess what would be the total # of comparisons if n=100. - In the case of a sorted list, selection sort requires the same number of steps as running through an unsorted list. The default implementation of selection sort is not stable; however, it can be made stable if required. To explore more about stable selection sort, refer to this [link](https://www.geeksforgeeks.org/stable-selection-sort/). ## Selection Vs Bubble Sort - In bubble sort, the adjacent element is compared and swapped, whereas in selection sort the smallest element is selected to swap against the largest element. - The selection sort has improved efficiency and is faster when compared to bubble sort. - The number of swaps is O(n) in comparison to О(*n^*2) of bubble sort. In our example , we have very few swaps when compared to swaps in bubble sort. - The worst-case complexity in both algorithms is О(*n^*2). In the case of an unsorted list, selection sort has to iterate over each of the ‘n’ elements to find the smallest number. One has to repeat ‘n’ times to sort out the array. - In the best case of an already sorted list, selection sort will need to iterate over all elements to check if it's sorted. Due to this reason, the time complexity remains the same as in the worst case О(*n^*2). - The best case of bubble sort is O(n) because in the first iteration, if the swaps are zero the array is determined to be completely sorted. - Bubble sort consumes additional space for storing variables and needs more swaps, whereas selection sort is an in-place algorithm, so it doesn’t need to allocate any memory. - Selection sort stores one minimum value and its index to compare results. The space required doesn’t increase with the input size, hence space complexity remains O(1). **Time and space complexity** are listed in the below table: ![Table comparing time and space complexity of selection sort versus bubble sort](../../assets/blog/reviewing-sorting-algorithms-selection-sort-2.png) ## Code Implementation: > [!NOTE] > The code has a couple of counters and log messages only for demonstration purposes. ```python #Function to sort array using Selection Sort def selectionSort(arr): # Get the length of the array arrlen = len(arr) print('Length of the Input Array : {}'.format(arrlen)) # Counter to count # of iterations cnt = 0 cnt_comps = 0 # count the comparisions for i in range(0, arrlen-1): # Initial Loop to iterate through Arrays # set the indexposition for example i = 0,1,2,3,4 indexpos = i # set the smallestValue as the Current index position array Value smallValue = arr[indexpos] # increment the iteration counter cnt = cnt+1 print('- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -') print('Iteration : {} starting array is {} '.format(cnt,arr)) # Loop to compare the index position value with rest of the values in array for j in range(i+1, arrlen): print('comparing {} and {} '.format( smallValue , arr[j])) cnt_comps = cnt_comps + 1 # Compare smallestValue with the rest of the values # This is to check for any values which is smaller than the current position set value. if(smallValue > arr[j]): # swap the indexposition to the index of newly found small value indexpos= j # Imp: Swap smallvalue with current array indexposition value. # This means the comparision value is changed. This works for large arrays with a complete unsorted list smallValue= arr[j] # if indexposition is not equal to original index position Swap the values if (indexpos!= i): currentvalue = arr[i] print('Swapping the number {} >> for the number {}'.format(arr[indexpos],arr[i])) arr[i], arr[indexpos] = arr[indexpos], currentvalue print('Current Array after swap : {} '.format(arr)) else: # else just print No swap is performed print('>>> Looks like value is sorted. No Swap is performed >>>') print('Total iterations :{} Total Comparisions : {} Output Array is : {}'.format(cnt,cnt_comps,arr)) return arr # Test Long unsorted list #arrlist = [3, 56, 2, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] # Test short unsorted list arrlist = [10,4,8,6,1] # Test sorted list arrlist = [1,2,3,4,5] print(selectionSort(arrlist)) ``` **Log Output:** Let’s decipher the log output to align with our understanding of selection sort. ```text # Log for Test short unsorted list arrlist = [10,4,8,6,1] Length of the Input Array : 5 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 1 starting array is [10, 4, 8, 6, 1] comparing 10 and 4 comparing 4 and 8 comparing 4 and 6 comparing 4 and 1 Swapping the number 1 >> for the number 10 Current Array after swap : [1, 4, 8, 6, 10] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 2 starting array is [1, 4, 8, 6, 10] comparing 4 and 8 comparing 4 and 6 comparing 4 and 10 >>> Looks like value is sorted. No Swap is performed >>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 3 starting array is [1, 4, 8, 6, 10] comparing 8 and 6 comparing 6 and 10 Swapping the number 6 >> for the number 8 Current Array after swap : [1, 4, 6, 8, 10] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 4 starting array is [1, 4, 6, 8, 10] comparing 8 and 10 >>> Looks like value is sorted. No Swap is performed >>> Total iterations :4 Total Comparisons : 10 Output Array is : [1, 4, 6, 8, 10] [1, 4, 6, 8, 10] ---*---*---*---*---*---*---*---*---*---*---*---*---*---*---*---*---*---*---*---* # Log for Test long unsorted list arrlist = [3, 56, 2, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] Length of the Input Array : 13 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 1 starting array is [3, 56, 2, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] comparing 3 and 56 comparing 3 and 2 comparing 2 and 58 comparing 2 and 79 comparing 2 and 59 comparing 2 and 34 comparing 2 and 23 comparing 2 and 4 comparing 2 and 78 comparing 2 and 8 comparing 2 and 123 comparing 2 and 45 Swapping the number 2 >> for the number 3 Current Array after swap : [2, 56, 3, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 2 starting array is [2, 56, 3, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] comparing 56 and 3 comparing 3 and 58 comparing 3 and 79 comparing 3 and 59 comparing 3 and 34 comparing 3 and 23 comparing 3 and 4 comparing 3 and 78 comparing 3 and 8 comparing 3 and 123 comparing 3 and 45 Swapping the number 3 >> for the number 56 Current Array after swap : [2, 3, 56, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 3 starting array is [2, 3, 56, 58, 79, 59, 34, 23, 4, 78, 8, 123, 45] comparing 56 and 58 comparing 56 and 79 comparing 56 and 59 comparing 56 and 34 comparing 34 and 23 comparing 23 and 4 comparing 4 and 78 comparing 4 and 8 comparing 4 and 123 comparing 4 and 45 Swapping the number 4 >> for the number 56 Current Array after swap : [2, 3, 4, 58, 79, 59, 34, 23, 56, 78, 8, 123, 45] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 4 starting array is [2, 3, 4, 58, 79, 59, 34, 23, 56, 78, 8, 123, 45] comparing 58 and 79 comparing 58 and 59 comparing 58 and 34 comparing 34 and 23 comparing 23 and 56 comparing 23 and 78 comparing 23 and 8 comparing 8 and 123 comparing 8 and 45 Swapping the number 8 >> for the number 58 Current Array after swap : [2, 3, 4, 8, 79, 59, 34, 23, 56, 78, 58, 123, 45] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 5 starting array is [2, 3, 4, 8, 79, 59, 34, 23, 56, 78, 58, 123, 45] comparing 79 and 59 comparing 59 and 34 comparing 34 and 23 comparing 23 and 56 comparing 23 and 78 comparing 23 and 58 comparing 23 and 123 comparing 23 and 45 Swapping the number 23 >> for the number 79 Current Array after swap : [2, 3, 4, 8, 23, 59, 34, 79, 56, 78, 58, 123, 45] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 6 starting array is [2, 3, 4, 8, 23, 59, 34, 79, 56, 78, 58, 123, 45] comparing 59 and 34 comparing 34 and 79 comparing 34 and 56 comparing 34 and 78 comparing 34 and 58 comparing 34 and 123 comparing 34 and 45 Swapping the number 34 >> for the number 59 Current Array after swap : [2, 3, 4, 8, 23, 34, 59, 79, 56, 78, 58, 123, 45] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 7 starting array is [2, 3, 4, 8, 23, 34, 59, 79, 56, 78, 58, 123, 45] comparing 59 and 79 comparing 59 and 56 comparing 56 and 78 comparing 56 and 58 comparing 56 and 123 comparing 56 and 45 Swapping the number 45 >> for the number 59 Current Array after swap : [2, 3, 4, 8, 23, 34, 45, 79, 56, 78, 58, 123, 59] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 8 starting array is [2, 3, 4, 8, 23, 34, 45, 79, 56, 78, 58, 123, 59] comparing 79 and 56 comparing 56 and 78 comparing 56 and 58 comparing 56 and 123 comparing 56 and 59 Swapping the number 56 >> for the number 79 Current Array after swap : [2, 3, 4, 8, 23, 34, 45, 56, 79, 78, 58, 123, 59] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 9 starting array is [2, 3, 4, 8, 23, 34, 45, 56, 79, 78, 58, 123, 59] comparing 79 and 78 comparing 78 and 58 comparing 58 and 123 comparing 58 and 59 Swapping the number 58 >> for the number 79 Current Array after swap : [2, 3, 4, 8, 23, 34, 45, 56, 58, 78, 79, 123, 59] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 10 starting array is [2, 3, 4, 8, 23, 34, 45, 56, 58, 78, 79, 123, 59] comparing 78 and 79 comparing 78 and 123 comparing 78 and 59 Swapping the number 59 >> for the number 78 Current Array after swap : [2, 3, 4, 8, 23, 34, 45, 56, 58, 59, 79, 123, 78] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 11 starting array is [2, 3, 4, 8, 23, 34, 45, 56, 58, 59, 79, 123, 78] comparing 79 and 123 comparing 79 and 78 Swapping the number 78 >> for the number 79 Current Array after swap : [2, 3, 4, 8, 23, 34, 45, 56, 58, 59, 78, 123, 79] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 12 starting array is [2, 3, 4, 8, 23, 34, 45, 56, 58, 59, 78, 123, 79] comparing 123 and 79 Swapping the number 79 >> for the number 123 Current Array after swap : [2, 3, 4, 8, 23, 34, 45, 56, 58, 59, 78, 79, 123] Total iterations :12 Total Comparisons : 78 Output Array is : [2, 3, 4, 8, 23, 34, 45, 56, 58, 59, 78, 79, 123] [2, 3, 4, 8, 23, 34, 45, 56, 58, 59, 78, 79, 123] ---*---*---*---*---*---*---*---*---*---*---*---*---*---*---*---*---*---*---*---* # Log for Test Sorted list arrlist = [1,2,3,4,5,6] Length of the Input Array : 5 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 1 starting array is [1, 2, 3, 4, 5] comparing 1 and 2 comparing 1 and 3 comparing 1 and 4 comparing 1 and 5 >>> Looks like value is sorted. No Swap is performed >>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 2 starting array is [1, 2, 3, 4, 5] comparing 2 and 3 comparing 2 and 4 comparing 2 and 5 >>> Looks like value is sorted. No Swap is performed >>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 3 starting array is [1, 2, 3, 4, 5] comparing 3 and 4 comparing 3 and 5 >>> Looks like value is sorted. No Swap is performed >>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Iteration : 4 starting array is [1, 2, 3, 4, 5] comparing 4 and 5 >>> Looks like value is sorted. No Swap is performed >>> Total iterations :4 Total Comparisons : 10 Output Array is : [1, 2, 3, 4, 5] [1, 2, 3, 4, 5] ``` If you analyze the short unsorted list [10,4,8,6,1] log, length of the array is 5, total iterations performed are 4 and total Comparisons 10. The total comparison value is based on our earlier formulae n(n-1) /2. Another test performed with a long unsorted list has log output of array length as n=13, total iterations performed (n-1) = 12, total comparisons as 78 based on the above formulae. If you recollect, we had to guess what would be the total comparisons for n=100, it will be 4950. We also tested a completely sorted array [1,2,3,4,5], although no swapping was required, the total number of comparisons remains 10 and iterations as 4. This clearly justifies the selection sort time complexity of О(*n^*2) in best and worst cases. ## Final thoughts Selection sort is yet another simple algorithm that is slow and not fast in comparison to bubble sort. However, it works well with a small list to be sorted and in cases where space is limited. This feature is attributed mainly due to the minimal swaps O(n) requiring very less space. To conclude, selection sort doesn’t seem to be the answer for the fast and efficient sorting algorithm we were looking for at the beginning of the post. However, this isn’t the end of it, in the continuum we have a couple more sorting algorithms to be discussed, to arrive at the most efficient sorting techniques. Stay tuned!! I hope you enjoyed the post. Please subscribe to [ProstDev](https://www.prostdev.com/) for more exciting topics. Hasta luego, amigos! ## Study Resources Below are some resources which discuss Selection sort and its Time-Space complexity at length. - [The Selection Sort - Runestone academy](https://runestone.academy/runestone/books/published/pythonds/SortSearch/TheSelectionSort.html) - [Selection Sort - Wikipedia](https://en.wikipedia.org/wiki/Selection_sort) - [Video] [Selection Sort - CS50 Harvard](https://www.youtube.com/watch?v=f8hXR_Hvybo&feature=emb_title) - [Analysis of Selection Sort - Khan Academy](https://www.khanacademy.org/computing/computer-science/algorithms/sorting-algorithms/a/analysis-of-selection-sort) - [Video] [Select Sort - CS310 San Diego University](https://www.youtube.com/watch?v=fgYlVyrt1vE) - [Video] [Selection Sort with Gypsy Folk Dance](https://www.youtube.com/watch?v=Ns4TPTC8whw) - [Asymptotic-Notation- Khan Academy](https://www.khanacademy.org/computing/computer-science/algorithms/asymptotic-notation/a/asymptotic-notation) - [Big-O cheat sheet](https://www.bigocheatsheet.com/) --- ## Software Quality is not cheap, but would you risk your company's future for the lack of it? Source: https://prostdev.com/post/software-quality-is-not-cheap-but-would-you-risk-your-company-s-future-for-the-lack-of-it | Published: Jul 9, 2020 | Category: Guides You have less than 6 minutes before your customer ditches your application because they perceived a bad customer experience. If you knew this, would you risk your company’s future on the lack of software quality? Mark is the CEO of a start-up company. He had 3 teams of developers and only one QA (Quality Assurance resource) to perform the vast amount of work before pushing the web application to go live (to make it available for customers). He had a healthy number of customers that were eager to try the new CRM (Customer Relationship Management) system. Little did everyone know; things went south very quickly. Customers weren’t happy because the web application wasn’t working as expected. There was a gap between what the application was supposed to do and what it actually did. So, what really went wrong? Short answer: Lack of quality and testing. Today we are going to talk about the importance of testing. We will also talk about some testing types and objectives but, first things first. ## What is Quality? There are 3 points of view to look at the software product. These views pay attention to different attributes of the software. **The first point of view concentrates on the functionality** of the software to conform to requirements. A few examples of this: - It uses a specification guide in the form of: - User Stories – In the agile world. - Specifications document. - The QA (Quality Assurance resource) will focus all their efforts into making sure the software solution provides the minimum specifications stated in the document they got assigned. **2. Best practices when coding.** A few examples of this: - Write reusable code (reutilization of existing code for new functions in the software) - Write unit tests for your unit of behavior (to make sure a piece of code behaves as it’s supposed to). - Make your code readable and self-documenting through good naming conventions and programming standards. - Not relying on code comments because they become outdated often. . (Software developers use comments to explain what their code performs) - Refactor (process of restructuring existing computer code) whenever you can or need. - Avoid technical debt (additional rework caused by choosing a limited solution now instead of using a better approach that would take longer). - Try to make your code fail a test at least once, that way you know that it works as expected. **3. The end user or customer.** A few examples of this: - Quality is a customer determination. - Exceed customer’s expectations. - Provide customer satisfaction. **In my opinion, good software quality could be defined as:** A software product that is free from defects, that is efficient, secure, reliable, easy to maintain, that has the ability to escalate with little effort and time and that exceeds customer’s expectations by providing a solution that is, not only useful, but adds more value to the customer. ## What is the importance of testing? I’ll use an analogy to try and explain the importance of it. It’s like throwing yourself from an airplane and you didn’t check if your backpack had a parachute. In this case you are the software product and the empty backpack is your product quality. From a business point of view, **software quality is your safety net and it saves you time and money.** **It brings you economic benefits** in the medium-long run. Every time a customer is not able to interact successfully with your software product, it becomes a customer that will never come back. When you lose that customer, you lose money and your software product might get a bad review. **Did you know that it takes around 40 positive reviews to undo only one negative review?** And most happy customers don’t leave a positive review. From a security point of view, software quality protects vulnerabilities of your product. That means that your product might be targeted to exploit those vulnerabilities in order to steal customers personal information, money, benefits-free products or anything else you might keep in order to process their software solution. **From a customer satisfaction point of view, software quality provides precisely that: Customer satisfaction.** The software product that you create will bring a polished solution and the best user experience possible (at that precise time). It will be efficient, secure, reliable and will work like magic. So easy to interact with that no one will ever know the complexity in the code. And you might be hesitant into investing in testing because it seems to be so invisible. No one cares when your software solution is working fine, right? But what happens when it doesn’t perform well? Software quality is not cheap when perceived as invisible but if you start adding $ into each defect found you soon will learn that it is not only necessary, it is intrinsic to your business success. **A software with good software quality, reputation and positive user experience will bring you more customers.** ## Software Quality Types and Objectives Software quality is divided into 2 main categories. Functional and non-functional. **Functional testing** is the **“what”** in your software. What will your application do? What should your TV do when you push the power button? - Turn on the screen - This is a functional test. How long should your TV take when you push the power button to turn on the screen? - 2 seconds - This is a non-functional test. **Non-functional testing** is the **“how”** in your software. There are more than 100 types of testing but some of the most common types are listed below. ![Diagram listing common functional and non-functional software testing types](../../assets/blog/software-quality-is-not-cheap-but-would-you-risk-your-company-s-future-for-the-lack-of-it-1.png) In software testing you have several opportunities to catch the defects in your product before launching it to your customers. **Unit Testing** is the first level or opportunity for software testing that is performed by software developers. It is the first piece of the puzzle. It is performed by using the White Box Testing method, which means that the person who is testing the functionality also knows the internal structure, design and implementation. The main objective is to isolate your unit and test its behavioral performance. Make sure it performs as designed. The second level or opportunity for testing is **Integration Testing**. Integration Testing combines individual units, like the one we talked about above, and tests them as a group. The idea behind this type of testing is to expose defects in the interaction between the units. An example of integration testing could be something similar to a lightbulb connected to a battery and a switch. Each one of the items listed is a unit, and I expect that when I interact with the switch, the lightbulb will turn on and off. The third level or opportunity to catch a defect comes in the form of **System Testing**. This means that we will combine all the product components and modules and perform an end-to-end testing on it. In this type of testing we use a persona and try to accomplish a task in the software product. There’s a beginning and an end to the workflow that has to be followed to get the task done. The objective is to replicate and validate real user scenarios. The fourth level is named **Acceptance Testing**. The software product is tested against the business requirements. There has to be a perfect compliance with the specifications on the solution we are presenting to the end user. Product quality is a joint effort. To have a mesmerizing quality you need everyone in your company to put on their Quality Assurance hat and evaluate their own designs, implementations and reasons behind each solution. A software tester (or QA) is your software solution protector. It is the last gate to open before launching your product to your customer. The difference between making it or breaking it. ## Recap Quality can be explained from 3 different perspectives: - The functionality of the software to conform to requirements. - Using best practices when coding the software product. - Exceed customer’s expectations. Testing your software product will result in more customer satisfaction, which can give you a great reputation as a business and will bring you more customers, resulting in more money. The types of Software Testing can be divided into Functional and Non-Functional. - Functional testing is the “what” in your software. - Non-Functional testing is the “how”. --- ## Scatter-Gather Integration Pattern (Mule 4) Part 2 Source: https://prostdev.com/post/scatter-gather-integration-pattern-mule-4-part-2 | Published: Jul 7, 2020 | Category: Guides *GitHub repository with the Mule Project can be found at the end of the post.* In the previous post, [Intro to Scatter-Gather Integration Pattern](https://www.prostdev.com/post/intro-to-scatter-gather-integration-pattern), we have discussed the basics of the Scatter-Gather message routing Enterprise Integration Pattern and also demonstrated a simple case of the pattern using MuleSoft Connector. Moving forward, from a happy path scenario, we will see the demonstration on how Mule 4 Scatter-Gather connector behaves in case of processing errors with or without Mule Error handlers. Additionally, we will explore the example of processing multiple services using the Mule 4 Scatter-Gather connector. **Gathering** **thoughts from the previous blog:** - Scatter-Gather processes messages in-parallel and the result is an aggregated response of the routes. - To live up to its merits, Scatter-Gather needs more than one message processing route. - When the process responses are independent of each other, it is efficient to use Scatter-Gather. - Use Scatter-Gather for multicasting scenarios. - Scatter-Gather is concurrent processing and there is no sequence of execution. - Scatter-Gather will not stop processing messages to other successful routes during failures. ## How does the input message get processed in Scatter-Gather? The input payload/message is distributed or copied across all the routes of the Scatter-Gather. Our [previous example](https://www.prostdev.com/post/intro-to-scatter-gather-integration-pattern) of the MuleSoft Scatter-Gather is modified, by adding a new Set Payload component to check the status of the payload during the Scatter-Gather processing. As shown below from the log results, the payload is copied or distributed across all the routes of the Scatter-Gather. In the listed example, the payload value “result0: 42” is not honored by the three flows, as they have their payload values. In short, the Set Payload value before the Scatter-Gather component is overwritten by the flow's payload value. ![Fig 2.0: Input payload is copied to all the route processors.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-1.png) However, the Set Payload information can be very well used to add business logic to the multiple routes accordingly. For demonstration purposes, Flow One payload is modified to retrieve the “result0” information. ![Fig 2.1: Initial Set Payload result is displayed to include in the flow-one response.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-2.png) ## Error Handling in Scatter-Gather In our [previous post](https://www.prostdev.com/post/intro-to-scatter-gather-integration-pattern), we stated that the Scatter-Gather avoids a complete meltdown, unlike sequential processing. But, without an effective error handling in place, you can question the merits of the latter statement. One can use a MuleSoft Try scope in the routers of a Scatter-Gather component for error handling purposes. If a particular route encounters errors, the Try scope will take care of the error using the [On-Error-continue](https://docs.mulesoft.com/mule-runtime/4.3/on-error-scope-concept) error handler, which would make the route successful by handling the error logic. The Scatter-Gather route failures are handled well to process the error unless it is required to end the process abruptly. Let’s try to prove the same using some twists in our current Scatter-Gather example. ### Happy Path ☺ Everything is green, life’s pretty smooth and happiness sways around. The Scatter-Gather route flows one, two, and three payloads have been modified to implement a simple addition, subtraction, and division of numbers (ex: 10+2, 16-2, 30/2) to be aggregated as an output result of each flow. In the screenshot below, results are displayed as expected, all looks good! Yay!! ![Fig 2.2: Result of the happy path Scatter-Gather flow without any errors.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-3.png) ### Not so Happy Path Well, life isn’t all rainbows and unicorns as the above result made you believe for a while. If the flow-three fails due to a *division by zero* error, the Scatter-Gather will fail as there isn’t any mechanism to avert the catastrophe. The process will halt, and the error will be displayed as given below. ![Fig 2.3: Mule debugger showing the Flow-Three failure due to error. No error handling is implemented.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-4.png) ![Fig 2.4: Postman test of the Scatter-Gather Flow-Three failure due to error.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-5.png) Although not so happy path the above case is, one would still want to see some result as an output instead of a long semi-cryptic error message. Here comes the error handling Try scope for the Scatter-Gather routes! All the three flows are wrapped with [Try Scope](https://docs.mulesoft.com/mule-runtime/4.3/try-scope-concept)to capture the errors in any of the routes if they failed while processing the message. The output is guaranteed to be some aggregated result instead of a long error description. ![Fig 2.5: Try-Scope error handling implemented in all three processes of the Scatter-Gather.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-6.png) From the above fig. 2.5, an error handling scope (Mule 4) of On-Error Continue is implemented which will make sure a result is derived from the Scatter-Gather. More information about Mule 4 error handling can be found at [MuleSoft documentation](https://docs.mulesoft.com/mule-runtime/4.3/intro-error-handlers). The transform message component, inside the On-Error Continue scope, is used to provide more information on the type of error as output. By adding this feature, we can now identify the failure reason for flow three. After all, our goal here is to achieve a more composed state of the output. There will be valid business scenarios in real-time, where the output of the route in error is not required to be displayed. In such cases, the output will be an aggregate response of successful flows, as shown in Fig 2.5.1. In the On-Error continue component, the transform message payload is given as [ ] (empty array) to bypass error information. ![Fig 2.5.1: Postman test of the Scatter-Gather flow with error handling implemented. Only success responses from the processes are displayed.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-7.png) Fig.2.6 is showing results of the flow stepping into the error handling scope through Anypoint studio mule debugger. The error is bypassed, and the Scatter-Gather has aggregated the messages from successful processes. ![Fig 2.6: Display of the Flow-Three error handling and log showing error information.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-8.png) When the flow-three result is required irrespective of failure or success, a descriptive user-friendly error message can be added. This would help the user to identify the reason for Scatter-Gather route failures (if any). If a business logic needs to be invoked on flow-three failure, it can be incorporated in the On-Error continue error handler scope. The business logic can be anything specific to invoking another process upon failure or notifying the stakeholders through email. ![Fig 2.7: Postman test of the Scatter-Gather flow with error handling implemented. All the route responses are displayed, including error.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-9.png) Error message to be displayed as the output of the failed process flow-three. ![Fig 2.8: Flow three error handling response returned.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-10.png) Yay!! We searched for happiness, in the not so happy path of Scatter-Gather and we were able to strike the right balance. A thought to ponder and reflect upon, which brings us to our reflection questions, and I would encourage you to try out the code snippets in mule 4 to try out the below cases and get the answers. ### Reflection Questions: - What happens to the initial payload, when one of the processes in the Scatter-Gather fails? - What happens if the variable in one process is modified and not in the other? - What would be the result of Scatter-Gather if “On-Error propagates” is used instead of “On-Error continue” when one of the routes fails? ## Multiple Services Call Using Mule 4 Scatter-Gather Okay! Moving ahead, remember we are yet to see a case of multiple services being invoked through the Scatter-Gather Mule 4 component. The example demonstrated below is retrieved from the free [Mule 4 developer](https://training.mulesoft.com/course/development-fundamentals-mule4) course. It’s a very simple example to showcase the power of the Scatter-Gather component. The scenario is to search the flights of United Airlines and Delta airlines for a given airport code. The United Airlines flight search is retrieved from REST API calls. Whereas Delta airlines information is retrieved through SOAP WSDL. The Set-Variable component is used to store the airport code value to retrieve the available flight information. Variable ‘code’ value is set as #[message.attributes.queryParams.code]. ![Fig 2.9: Scatter-Gather example showing multiple services (HTTP REST, SOAP WS) invoked.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-11.png) The following result is derived from the Scatter-Gather processing of the two different service calls. ![Fig 2.10: Postman test displaying the results of United and Delta flights.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-12.png) You have made it to this part of the post, now the last stop in the journey is to add the missing part in the above demonstration, error handling. If you check *Fig 2.9,* Try scope hasn’t been implemented yet. Applying the learnings from our previous section, let’s add the Try scope to each of the Scatter-Gather routes. We can add a flavor to the scenario by displaying only available flights. Meaning, we are looking to retrieve any flight information irrespective of route failure. You can most definitely hash out a test case to check the output if both the flights fail. To demonstrate the error handling, the web services URL is tweaked as [http://mu.learn.mulesoft.com/delta12?wsdl](http://mu.learn.mulesoft.com/delta12?wsdl) ![Fig 2.11: Implemented Try-scope to all the processes.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-13.png) Upon checking the flow through mule-debugger, we can see the getDeltaFlights process failed due to error type `WSC: CONNECTIVITY`. ![Fig 2.12: getDeltaFlights process failure due to connectivity.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-14.png) However, the failure was handled successfully by the MuleSoft Try scope and the On-Error continue to Transform message has an empty array returned as a response. Scatter-Gather has now aggregated the response from both the flows as output shown below: ![Fig 2.13: Postman test of the Scatter-Gather flow with the final response from Scatter-Gather with only United flights.](../../assets/blog/scatter-gather-integration-pattern-mule-4-part-2-15.png) ## Penultimate Thoughts Finally, we have reached a point where we understand the ball bearings of Scatter-Gather through the Mule 4 connector example. To reiterate: - Scatter-Gather is best suited for Multicast scenarios. - The Scatter-Gather messages are processed in-parallel with aggregate responses. - More than one process is required for Scatter-Gather to function according to its discussed merits. - The MuleSoft Scatter-Gather has many features to control the execution flow in case of route failures. - In case of errors, a new business logic such as notifications can be invoked as required. Now that you’re familiar with Scatter-Gather, it’s the time to check out the real code in action. ### Configuration File Code: ```xml { airlineName: flight.airlineName, availableSeats: flight.emptySeats, departureDate: flight.departureDate, destination: flight.destination, flightCode: flight.code, origination: flight.origin, planeType: flight.planeType, price: flight.price } as Object { class : "com.mulesoft.training.Flight" }]]> { airlineName: return.airlineName, availableSeats: return.emptySeats, departureDate: return.departureDate, destination: return.destination, flightCode: return.code, origination: return.origin, planeType: return.planeType, price: return.price } as Object { class : "com.mulesoft.training.Flight" }]]> ``` ## Next Steps It is highly recommended that you check the simple example code snippets and test out the concepts discussed in these pair of blogs. Download [MuleSoft Anypoint Studio](https://www.mulesoft.com/lp/dl/studio) to test out the Scatter-Gather flows by goofing around in your unique ways. Test the above reflection questions to grasp an in-depth understanding of the Mule 4 Scatter-Gather component. Try out some [DataWeave](https://docs.mulesoft.com/mule-runtime/4.3/dataweave-cookbook) functions to make the error response interesting for the end-users. For a better understanding of the Mule 4 error handling refer to the below amazing blogs: - [Mule 4 error handling demystified](https://blogs.mulesoft.com/dev/training-dev/mule4-error-handling/) - [Mule 4 error handling deep dive](https://blogs.mulesoft.com/dev/training-dev/mule4-error-handling-deep-dive/) - [Use case-specific error handling in Mule 4](https://blogs.mulesoft.com/dev/training-dev/mule4-error-handling-use-cases/) I hope you enjoyed the post. Please subscribe to ProstDev for more exciting topics. Hasta luego, amigos! ## GitHub repository [ProstDev GitHub - Scatter-Gather Error Handling Example](https://github.com/ProstDev/apdev-scatter-gather-error-handling) --- ## Reviewing Sorting Algorithms: Bubble Sort Source: https://prostdev.com/post/reviewing-sorting-algorithms-bubble-sort | Published: Jun 30, 2020 | Category: Guides Let’s sort it out! In this series of posts, we will be discussing basic computer science algorithm sorting techniques. Yes, you are right we are here again trying to shake up our memory boxes for these algorithms long forgotten. Recently, I attended an interview for a Silicon Valley company, and you guessed it right, KO (Knock Out) coding rounds were the appetizers and I figured pretty much on my day 1 of coding interview preparation, that I'm not serving KO’s back on coding challenges. You may most definitely choose to agree with me that we always remember the most popular and unpopular of everything. I was sure the most unpopular sorting technique was “Bubble Sort” and the popular one was “Quick Sort”. Basically, my statement is incorrect on many counts about the popularity of the sorting techniques, the most unpopular kid in your high school is pretty much popular for being not so. ## Why is sorting a big deal? We know sorting things almost in every instance reduces the complexity of the problem at hand. Sorting is a well-solved problem in computer science, It puts any list coming out of a computer into a certain required order efficiently to reduce the searching complexity of the problem. In a series of posts, we will be discussing some of the sorting algorithms listed in the below order: - **Bubble Sort** - [Selection Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-selection-sort) - [Insertion Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-insertion-sort) - [Merge Sort](https://www.prostdev.com/post/reviewing-sorting-algorithms-merge-sort) - Quick Sort - Heap Sort Let’s start with the most popularly known Sorting technique. You are getting better at the game of *“who’s more popular!”* ## Bubble Sort Bubble sort is a very simple comparison-based algorithm used to sort the elements which are out of order. It compares the two elements at a time to check for the correct order, if the elements are out of order they are swapped. The swapping of elements continues, until an entire sorted order is achieved. Although a naïve algorithm, knowing bubble sort helps in understanding the technicalities of sorting in comparison with other techniques. Let’s play with the bubbles. **Problem:** Sort the numbers 10, 4, 8, 6, 1. As mentioned earlier, bubble sort will compare the two pairs at a time. Let’s start with the *first iteration* of the numbers: ![First bubble sort pass over [10, 4, 8, 6, 1], swapping pairs until 10 reaches the end](../../assets/blog/reviewing-sorting-algorithms-bubble-sort-1.png) Rinse and repeat! Until the required sorting order is achieved. This begs the question, what if the problem has a very long list of elements to be sorted like for example 100 or 1000? Well, the number of iterations to check every pair of elements repetitively might exhaust you (n-1) times. Which means, it will take (n-1) iterations in order to sort a list of n elements. *Mind you, the above example was just the very first iteration and for* *n=5**, it takes a total of 4 iterations to complete the sorting.* *2nd Iteration:* ![Second bubble sort pass comparing pairs, leaving 8 and 10 sorted at the end](../../assets/blog/reviewing-sorting-algorithms-bubble-sort-2.png) *3rd iteration:* ![Third bubble sort pass, swapping 6 and 1 so 6, 8, 10 are now in order](../../assets/blog/reviewing-sorting-algorithms-bubble-sort-3.png) *4th iteration:* ![Fourth bubble sort pass swaps 4 and 1, producing the fully sorted [1, 4, 6, 8, 10]](../../assets/blog/reviewing-sorting-algorithms-bubble-sort-4.png) **Observations based on Iterations:** - *‘n’* number of elements will undergo (n-1) iterations to complete the sort. - If you have noticed, the max number in the list 10 is placed in its ordered position after the first iteration. - Likewise, the next max number 8 will be placed in its sorted position after the 2nd iteration, 6 from the 3rd iteration and number 4 for the 4th iteration. - Based on this pattern, we can see that in each iteration the largest element tends to move up in the correct order similarly to bubbles rising on the surface or the smaller elements bubbles up to top the list. Hence, the name suffices the feature as “Bubble Sort”. - We also observed that it's totally not required to check the last pair after the 2nd iteration as they are already sorted in their place. This pattern is observed in the 3rd iteration too, where the last 3 elements are already sorted. Which means that for every ‘m’ iterations, checking last ‘m’ elements will be a repetitive activity and avoiding the check can result in an optimal bubble sorting solution. - Hold on to these observations for now, as we still have to check why this kid on the sorting block is notoriously ‘unpopular’. ## Coding Demonstration of Bubble Sort using Python: ```python # Function to demonstrate bubble sort technique def bubbleSort(alist): length = len(alist) # Counter is used only for displaying iteration count cnt = 0 for i in range(length): # Check the element pairs for correct Order. 'n' elements will yield n-1 iterations to achieve sorted list cnt = cnt+1 isSwapped = False print('- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -') print(" Iteration : {} starting array is {} ".format(cnt,alist)) for j in range(length-i-1): # now, compare the elements to probe the order. Example : First element pair is compared print ("Now Comparing >> {} with {} : ".format(alist[j],alist[j+1])) if alist[j] > alist[j+1]: # swap the elements. For clarity purposes the swapping is shown in seperate steps # Use a Temp variabe to store the first element number in a element pair ex (1,2), temp will store value 1 # Temp variable is not required for swapping, is used only for understanding the swap logic temp = alist[j] # swap first element in pair with second element alist[j] = alist[j+1] # swap second element in pair with first element alist[j+1]= temp # As the element pair is sorted set it to true isSwapped = True if isSwapped : print(" Array is modified as {} ".format(alist)) else: print(" Array is not modified {} ".format(alist)) # Increase the counter to process into next element pair print( 'Iteration {} ending array {} : '.format(cnt,alist)) print( ' is array Swapped at least Once in the Iteration >> {} '.format(isSwapped)) if isSwapped == False: break return alist alist = [10,4,8,6,1] print(bubbleSort(alist)) ``` **Log Output:** ![Python console log of bubble sort iterations 1 and 2 with each comparison and swap](../../assets/blog/reviewing-sorting-algorithms-bubble-sort-5.png) ![Python console log of bubble sort iterations 3, 4 and start of 5](../../assets/blog/reviewing-sorting-algorithms-bubble-sort-6.png) ![Final log line: iteration 5 makes no swaps, confirming [1, 4, 6, 8, 10] is sorted](../../assets/blog/reviewing-sorting-algorithms-bubble-sort-7.png) From the above code snippet, we have demonstrated our earlier example on how bubble sort bubbles up the largest number to the end of the list in each iteration. What if the Incoming list is already sorted? In this case, if there are no swaps performed in the first iteration, it's positive that the list is pretty much already sorted out. ![Python bubble sort code beside its log, where an already-sorted list ends after one pass](../../assets/blog/reviewing-sorting-algorithms-bubble-sort-8.png) ## Time and Space complexity AKA why is this kid so Unpopular? To achieve the sorted list, we had to iterate ‘n’ times through an array of ‘n’ elements, in order to check all the ‘n’ elements. Meaning, the bubble sort performs *‘n*’ iteration *‘n’* times making it n*n = n^2. *But what is Time and space complexity? Why does it matter in this case?* - In simple terms, time and space complexity analysis is required to analyze the resources required by an algorithm to complete the function of the input (ex: in our case sorting). - Time complexity of an algorithm is the amount of time taken by the algorithm to complete processing as a function for given input length *n*. It is commonly expressed using asymptotic notations as given below table. - Space complexity is the amount of runtime memory used by the algorithm to achieve the result. Total space includes auxiliary space and the space used by the input. Auxiliary space is the temporary space allocated by the given algorithm so achieve the result for a given input size. Overall, memory usage includes the memory used to save the compiled version of instructions, space used by variables and constants and in cases where algorithms function internally another algorithm function. ![Table of bubble sort worst, best, average time complexity and space complexity](../../assets/blog/reviewing-sorting-algorithms-bubble-sort-9.png) ![Big-O complexity chart plotting operations vs elements from excellent to horrible](../../assets/blog/reviewing-sorting-algorithms-bubble-sort-10.png) *Fig: Reference -* [bigocheatsheet.com](http://bigocheatsheet.com) Best case time complexity: If the array is an already sorted list of elements, at most ‘n’ comparisons are made and no swaps are required. For example, input: [2,3,6,7,9,12] Worst case time complexity: Array is fully unsorted or in descending order. Example: [10,4,8,6,1] Average case time complexity: A mix of sorted and unsorted list of elements in the array. Example: [19,15, 21,7,9,10] Space complexity: Amount of memory required to run the algorithm grows no more than O(n). In our example, total space required would be for variables assigned and array size *n*. To understand why bubble sort is the least preferred sorting technique, consider the input list is grown in size to a more significant number (n=1000), the amount of comparisons and swaps will increase significantly making it more inefficient. I believe now it’s clear why bubble sort is so unpopular, as there are other sorting techniques which have a better time complexity and can run much faster for an increasing input size. To that point, bubble sort is mainly used for educational purposes. Stay tuned!! I hope you enjoyed the post. Please subscribe to [ProstDev](https://www.prostdev.com/) for more exciting topics. Hasta luego, amigos! ## Study Resources As bubble sort is a well known algorithm, lots of resources can be found with varying examples on the internet. This blog post was written to provide a basic understanding of the bubble sort, for a short read to refresh your memory. Below are some resources which discuss bubble sort and Time-Space complexity. - [The Bubble Sort - Runestone Academy](https://runestone.academy/runestone/books/published/pythonds/SortSearch/TheBubbleSort.html) - [Bubble Sort Wiki](https://en.wikipedia.org/wiki/Bubble_sort) - [Bubble Sort - CS50 Harvard](https://www.youtube.com/watch?v=8Kp-8OGwphY) - [Asymptotic-Notation- Khan Academy](https://www.khanacademy.org/computing/computer-science/algorithms/asymptotic-notation/a/asymptotic-notation) - [Big-O cheat sheet](https://www.bigocheatsheet.com/) --- ## 5 reasons why you need an IDE (and how it can save you so much time) Source: https://prostdev.com/post/5-reasons-why-you-need-an-ide | Published: Jun 25, 2020 | Category: Guides If you’re an experienced developer, you probably already know why using an IDE is a great idea, but if you’re completely new to the Software Development world, you may not even know what an IDE is. Well, not to worry! For this post, I will explain what is an IDE and five reasons why you need to start using one. I will also give you examples of my favorite IDEs for different programming languages and technologies. ## What is an IDE? IDE is an acronym for “Integrated Development Environment”, and it is nothing more than an application that can be used to develop any kind of software programs. You can usually download an IDE from the IDE’s creators’ website and install it like any other application; for example: for Windows, using a .exe file, or for MacOS, using a .dmg file. IDEs are designed to aid the programmers in their development process, and help them to be the most productive as possible by providing a Graphical User Interface (GUI) that can be easily used to perform basic day-to-day functions. A metaphor that comes to my mind is a calculator - if you’re a mathematician, you probably have bigger problems to solve than a basic average or division calculation; you can be more productive and solve more complex problems, if you don’t have to worry about the smallest operations that a tool can solve for you. This is basically what an IDE does for a Developer. If your idea of an IDE is not completely clear yet, with the next five reasons you’ll get a better understanding on what an IDE can do for you as a Software Developer. ### 1. Syntax highlighting If you speak english, you may say “cheers” after a toast at a wedding; but if you speak spanish, you’ll say “salud”; or you’ll say “prost” in german. Just like a regular language, every programming language has their own syntax, or their own way of “writing” or “communicating” the same thing. Here are some examples of how you would create two variables (x and y) and add them to create a new variable (z), in different programming languages: ![Same x + y addition written in plain Java, JavaScript, Python and DataWeave](../../assets/blog/5-reasons-why-you-need-an-ide-1.png) As you can see, they have some things in common, like the equal (=) or plus (+) symbols, but they also have some differences, like the end of a line (;) or the variable declarations (*int, var*). When you’re using an IDE that knows the programming language’s syntax that you’re using, the IDE is able to recognize what a symbol or a keyword means, and then it shows your code with certain colors or formats (like **bold** or *italic*) in order to make it a bit more readable. Here are the same previous examples, but using a different IDE for each of them: ![The same snippets with syntax highlighting in IntelliJ, Sublime Text, PyCharm and Anypoint Studio](../../assets/blog/5-reasons-why-you-need-an-ide-2.png) Note that the one for JavaScript (top-right corner) does not specify an IDE, but an Editor. Some text editors also offer syntax highlighting, just like an IDE would. However, an editor won’t give you the rest of the benefits (listed below) and if they do, it’s not going to be as efficient as an IDE. ## 2. Text autocompletion You know that thing Google does when you start writing some text for a new search and it wants to *complete* or give suggestions as to what you might be searching for? ![Google search box autocompleting the query technology blog with suggestions](../../assets/blog/5-reasons-why-you-need-an-ide-3.png) The same thing happens with an IDE. Since it already knows the syntax of the language you're programming in, it can give you suggestions of what you want to write next. This helps you to be faster because you don’t have to write everything yourself, you can just start writing something and the autocompletion will give you a list of possible choices! ![Animation of IntelliJ autocompleting a Java String keyword as the developer types](../../assets/blog/5-reasons-why-you-need-an-ide-4.gif) For example, say you forget a specific keyword that you need to use, but you do remember part of the instruction that is needed; instead of going to Google or to check on a programming book, you can just start writing what you remember, and the IDE will try to guess the command, then you just need to choose between your options and press Enter to select it. ![Animation of an IDE suggesting Java method calls like setDefaultCloseOperation as you type](../../assets/blog/5-reasons-why-you-need-an-ide-5.gif) Pretty cool, right?! ## 3. Refactoring options “Woah, wait! What does *refactoring* mean?” It means that you can restructure your existing code or project’s resources without affecting any behavior. For example: renaming a file, changing files to different folders, or even renaming a variable. “Ok, why would I want to do that if it already works as it is?” Sometimes the code can get too complex to understand, even by the developer who wrote it, and there are some changes that can be done to improve the code’s readability. For example, you create a function that will add two numbers and return the result, but you don’t want to think in descriptive names for the function or for the two variables, so you just named them “x”, “a” and “b”. This code will look like this: ```python def x(a, b): return a + b x(1, 2) ``` Not too readable, right? I have to go into the definition of the function “x” to know that the function is adding “a” plus “b”. Maybe if this function was named something like “add_numbers”, I wouldn’t have to waste my time going into the code to figure out that it’s adding two numbers. Instead of manually renaming the “x” in the definition of the function, and then once more in the function call (the last line), I can just select the “x”, right-click on it, and select “Rename”. ![PyCharm right-click context menu with Refactor and Rename highlighted on the function x](../../assets/blog/5-reasons-why-you-need-an-ide-6.png) After writing the new name for my function, I will see that all the references were correctly updated. This can also be done for the “a” and “b” variables, that can have a more descriptive name, like “first_number” and “second_number”. ![The function refactored to add_numbers(first_number, second_number) with all references updated](../../assets/blog/5-reasons-why-you-need-an-ide-7.png) Way better! Now I can clearly know what the “add_numbers” function will be doing. Without even going to the function’s definition, I can know in advance that the result of add_numbers(1, 2) will be 3. Now, this was a small example, but imagine that you have more than 20 files that are referencing the same function that you want to rename. Instead of manually going to each of the files, you can just use the IDE’s refactoring options to change the name. The same applies when moving files from one folder to another, or when renaming files. ## 4. Importing libraries When you’re working on a Software Development project, there will be some tough requirements that you need to figure out how to code. Luckily for us, you don’t have to create code from scratch, another developer on the internet already has and you can use their code as a new library. Again with my metaphors: imagine that you want to paint a house, but you don’t have any paint. Will you go to a forest to get some berries and insects to make your own paint? Or will you go to Home Depot and buy whatever paint you need? (I hope you chose the second one!). The same applies with code. There are some functions that someone created that are available for you to re-use in your project; there’s absolutely no need for you to reinvent the wheel in your code if you can just import a library and take advantage of it! There are times when you remember a library’s function, but you don’t remember the name of the library. So, same as before, instead of taking time off your day to google the library, you can just write the function that you wanted to use and let the IDE handle the import for you! ![Animation of an IDE auto-importing the library for Swing classes like JFrame and JButton](../../assets/blog/5-reasons-why-you-need-an-ide-8.gif) Are you convinced now? Yes? No? Maybe? Ok, let me give you the last reason why you need and want an IDE. ## 5. Build, compile, or run Once you finish coding your project or feature, you’ll want to do a quick test to verify that it’s working as you expected. How are you going to run your code in order to check that? Well, if you generated a bunch of Java code, you have to first *compile* your .java files into .class files (these are binary files that are interpreted by the computer), only after you do that, you can run your code. To do this, you have to do something in the Console (or Terminal) that looks like this (you don’t have to understand what I’m doing here, just notice there are some statements to run): ![Terminal manually compiling with javac then running java Main, printing Hello World!](../../assets/blog/5-reasons-why-you-need-an-ide-9.png) What if I told you that you can just click a button if you use an IDE? ![IntelliJ green-arrow menu with Run 'Main' option for the Hello World class](../../assets/blog/5-reasons-why-you-need-an-ide-10.png) Yes! Instead of doing all those commands manually, you can just use the “Run” button. You get exactly the same result! ![IntelliJ project tree and editor with the Run console printing Hello World! at the bottom](../../assets/blog/5-reasons-why-you-need-an-ide-11.png) If you need to build, compile, or run your code; you can do all these things just by clicking on a button from the IDE. No need to compile into a different format! (this can be time consuming). ### Recap To recap what we learned today, an IDE saves you so much time (and headaches) and we went through 5 ways that do this. - **Syntax highlighting** that helps you be able to read the code and not get lost in between hundreds of lines of code. The IDE uses different colors or formatting for the code, to make it more readable. - **Text autocompletion**, just like the Google search bar does, but for your code. If you forget a keyword, the IDE will give you the best suggestion of what you were looking for. - **Refactoring options**, like renaming a file, a function, or a variable; or moving files from one folder to another. This way you don’t have to manually update every single reference in the whole project. - **Importing libraries** in case you remember the code, but not the name of the library. Most likely the IDE already knows what function you’re referring to, it just has to import that library to your project and you’re all set. - **Build, compile, or run** your project just by clicking one button. There’s no need to memorize all those commands and run them one by one from the Console or Terminal. There are not only 5 benefits that an IDE can give you; there are a lot more that I love! Debugging, testing, generating pre-defined code, managing your dependencies, managing environments, using version control systems are all options that are more advanced. If you’re a new developer, or just new to the technology world, these 5 ways to save time are the most important ones that you need to know. What is your favorite IDE? Do you prefer editors over IDEs? Log-in now to leave me a comment about it! I’d love to hear your opinion :) *Prost!* -Alex ## More Links - [Download IntelliJ (Community Edition)](https://www.jetbrains.com/idea/download/) - [Download PyCharm (Community Edition)](https://www.jetbrains.com/pycharm/download/) - [Download Sublime Text](https://www.sublimetext.com/) - [Download Anypoint Studio](https://www.mulesoft.com/lp/dl/studio) --- ## Intro to Scatter-Gather Integration Pattern Source: https://prostdev.com/post/intro-to-scatter-gather-integration-pattern | Published: Jun 23, 2020 | Category: Guides *GitHub repository with the Mule Project can be found at the end of the post.* In this blog post we will be *gathering* the *scattered* thoughts by routing them into a singular direction. You would have probably guessed it by now, we will be discussing an Enterprise Integration Pattern for Message Routing called the Scatter-Gather. We will first try to understand the concept of Scatter-Gather and then demonstrate its ability with the help of MuleSoft ESB connector for the same. The goal of this post is to follow a very basic and simplistic approach, before diving further into the concept. If you are someone who doesn’t like a lot of text, the depicted pictures will whisper the secret, in parallel. ## What is Scatter-Gather? As described in [Enterprise Integration Patterns,](https://www.enterpriseintegrationpatterns.com/patterns/messaging/BroadcastAggregate.html) Scatter-Gather is a Message routing pattern which broadcasts messages to multiple recipients and aggregates the response back in a single message. In simple words, the messages are executed in parallel (scatter) and response from each execution is bundled (gathered) as one single message. Below picture depicts the same regarding Scatter-Gather: ![Fig 1.1: Request is routed to three different processes in scatter-gather, Output response will be the aggregated result of all 3 processes.](../../assets/blog/intro-to-scatter-gather-integration-pattern-1.png) > [!NOTE] > Response of individual processes are independent of each other. ## When to use Scatter-Gather? Depends on what you are trying to achieve, if it’s a [Fork-Joined model](https://en.wikipedia.org/wiki/Fork%E2%80%93join_model) (imagine a fork) or Multicast scenario, where the messages are required to be communicated simultaneously, Scatter-Gather is a better choice, in case of “[Fire-Forget”](https://www.enterpriseintegrationpatterns.com/patterns/conversation/FireAndForget.html) model use asynchronous processing blocks, as there is no standard requirement to gather all the response in given time. If performance is a parameter (most certainly), Scatter- Gather pattern helps to immensely cut down the processing duration and improves the performance. ## How does Scatter-Gather work? As already explained in the above section, Scatter-Gather will execute the messages in Parallel a.k.a. concurrent processing, meaning sending the messages to desired routes at the same time through a parallel thread pool. While a particular message is being processed, the remaining messages should wait for the completion of the message. In other words, the Scatter-Gather processing will be completed only when all the messages are fully processed. ![Fig 1.2: Scatter-Gather process 1 is complete (Green), Other two processes are work in-progress (Yellow). The response is not yet captured (Red).](../../assets/blog/intro-to-scatter-gather-integration-pattern-2.png) ![Fig 1.3: Scatter-Gather process 1 and 2 are complete (Green), Whereas process 3 is work in-progress (Yellow). The response is not yet captured (Red).](../../assets/blog/intro-to-scatter-gather-integration-pattern-3.png) ![Fig 1.4: Scatter-Gather process 1, 2, and 3 are complete (All Green. Yay!). The response is captured as a bundle of all three responses (Green).](../../assets/blog/intro-to-scatter-gather-integration-pattern-4.png) ### Quiz 1: What is the Total time taken for the completion of the Scatter-Gather process below? ![Scatter-gather diagram with three processes taking 20, 14, and 10 seconds, asking the total time.](../../assets/blog/intro-to-scatter-gather-integration-pattern-5.png) ## Why use the Scatter-Gather Integration Pattern? It may turn out to be a simple decision on why to use the Scatter-Gather pattern when compared to sequential processing, although the responses retrieved can be the same in both cases. Sequential processing should be considered when there is an interdependency of the processes up on their responses, meaning response from process 1 is required for process 2 to complete. In cases where the process responses are independent of each other, parallel processing can be efficiently executed. If using sequential processing, in case of one message/process failure the complete processing will come to halt. This may be a valid business scenario to end the process, whereas in case of scatter-gather the responses will include or exclude the error cases (scenario based) therefore avoiding a complete meltdown. ### Quiz 2: Scatter-Gather halts the processing of the messages if any one of the routing processes fail. *Answer: True or False.* We started with scattered thoughts about the pattern and now we have reached a state of composed thoughts, which were pretty *scattered* to begin with. Next, we will attempt to demonstrate our understanding through a MuleSoft (Mule 4) usage of Scatter-Gather Component. I won’t go deep into the specifics of Mule 4 Scatter-Gather Router as it has been explained very well in [MuleSoft documentation](https://docs.mulesoft.com/mule-runtime/4.3/scatter-gather-concept). ## Mule 4 Scatter-Gather Connector Example In the below example, a mule Scatter-Gather router invokes the three flows concurrently and aggregates the response as a new payload. *A point to note is Scatter-Gather block should have at least two routes to process.* The combined response is captured through Mule Transform Message component, where-in we flatten the payload to an array object. The output resultant payload of Scatter-Gather is an Object of Object. ![Mule flow with a Scatter-Gather routing to three Flow References, then a Transform Message.](../../assets/blog/intro-to-scatter-gather-integration-pattern-6.png) ![Transform Message showing the Scatter-Gather Object payload flattened with flatten(payload..payload).](../../assets/blog/intro-to-scatter-gather-integration-pattern-7.png) **Scatter-Gather Output Payload:** ![Scatter-Gather output index "0" holding the FlowOneResult payload plus its HTTP attributes.](../../assets/blog/intro-to-scatter-gather-integration-pattern-8.png) ![Scatter-Gather output index "1" beginning, showing its empty attachment and property name arrays.](../../assets/blog/intro-to-scatter-gather-integration-pattern-9.png) ![Scatter-Gather output index "1" holding the FlowTwoResult payload plus its HTTP attributes.](../../assets/blog/intro-to-scatter-gather-integration-pattern-10.png) ![Scatter-Gather output index "2" holding the FlowThreeResult payload plus its HTTP attributes.](../../assets/blog/intro-to-scatter-gather-integration-pattern-11.png) ![Continuation of the index "2" entry showing its GET method and TestScatterGather request paths.](../../assets/blog/intro-to-scatter-gather-integration-pattern-12.png) The payload size is 3 after the Scatter-Gather processing is complete. ![Mule Debugger showing the Scatter-Gather payload as a LinkedHashMap of size 3.](../../assets/blog/intro-to-scatter-gather-integration-pattern-13.png) ### Result of Scatter-Gather: The flow has been tested using POSTMAN. ![Postman GET to localhost:8081/TestScatterGather returning the aggregated result1, result2, result3 array.](../../assets/blog/intro-to-scatter-gather-integration-pattern-14.png) The debug process shows that the Scatter-Gather output payload has been aggregated (remember, we have flattened the payload to make it an array object). ![Mule Debugger showing the flattened payload as an array of the three flow results after Transform Message.](../../assets/blog/intro-to-scatter-gather-integration-pattern-15.png) The example shown above is a very simple case of implementing and understanding Scatter-Gather Pattern. In the [next post](https://www.prostdev.com/post/scatter-gather-integration-pattern-mule-4-part-2), we shall add more twists with different service call routings and discuss the topic of error handling in case Mule 4 Scatter-Gather. Until then, try to find your reflections of the *scattered* thoughts. Oh wait! The quiz? Yes! The answer to the first quiz is **20 seconds** and the second is **False**; Scatter-Gather doesn’t halt the processes abruptly with an effective error handling in place, indeed it will gather all the route information including the route which has errored and aggregates the results as output. In order to retrieve only successful route payload and ignore errors from Scatter-Gather processing, this can be efficiently handled using Mule 4 error handling process. We shall discuss the Scatter-Gather error handling cases in our next post. I hope you enjoyed the post. Please subscribe to ProstDev for more exciting topics. Hasta luego, amigos! ## References - [Enterprise Integration Patterns - Fire-and-Forget](https://www.enterpriseintegrationpatterns.com/patterns/conversation/FireAndForget.html) - [Enterprise Integration Patterns - Scatter-Gather](https://www.enterpriseintegrationpatterns.com/patterns/messaging/BroadcastAggregate.html) - [Enterprise Integration Patterns Using Mule](https://docs.mulesoft.com/mule-runtime/4.3/understanding-enterprise-integration-patterns-using-mule) - [Mule Docs - Scatter-Gather Router](https://docs.mulesoft.com/mule-runtime/4.3/scatter-gather-concept) ## GitHub repository [ProstDev GitHub - Scatter-Gather Example](https://github.com/ProstDev/apdev-scatter-gather-example) --- ## JSON Module Component in Mule 4 Source: https://prostdev.com/post/json-module-component-in-mule-4 | Published: Jun 16, 2020 | Category: Tutorials *GitHub repository with the Mule Project can be found at the end of the post.* The JSON Module Connector is used to validate a JSON Payload against the predefined JSON Schema of your choice. Since schema validation falls outside the scope of DataWeave Functionality, the JSON Module can be used. To use the JSON module, add it manually to your Mule app in Anypoint Studio or Design Center; or, if you want to use Maven, you can add the following dependency into your `pom.xml` file: ```xml org.mule.modules mule-json-module RELEASE mule-plugin ``` For better understanding of how the JSON Module works, I did a proof of Concept. Below is the screenshot of the flow which includes the JSON Module component. ![Anypoint Studio flow: Scheduler, Read, Logger, Validate schema, plus an On Error Propagate handler](../../assets/blog/json-module-component-in-mule-4-1.png) To validate the JSON payload, I need an example JSON file (in my case, `document.json`) and I would need the JSON schema file (`mySchema.json`) to validate my example JSON file against it. Since I chose to read the `document.json` file, I placed it below `src/main/resources/examples`: ![Project tree under src/main/resources showing document.json in examples and mySchema.json in schemas](../../assets/blog/json-module-component-in-mule-4-2.png) The `document.json` file looks like this: ![document.json array of employee records with some empty employeeID and employeeName values](../../assets/blog/json-module-component-in-mule-4-3.png) The schema against which the `document.json` file would be validated is defined as below: ![mySchema.json defining an array of objects requiring employeeID and employeeName with length limits](../../assets/blog/json-module-component-in-mule-4-4.png) A fixed frequency scheduler is set to run the flow every few seconds/minutes. The `document.json` file is read using the File Read operation as shown below: ![File Read operation configuration with File Path set to examples/document.json](../../assets/blog/json-module-component-in-mule-4-5.png) Once the file is read, the payload is saved in the Target Variable *jsonDoc* as shown below: ![Read operation Advanced tab with Target Variable set to jsonDoc and Target Value to payload](../../assets/blog/json-module-component-in-mule-4-6.png) After the file is saved in the *jsonDoc* variable, it is validated against the schema using the JSON Module as shown below: ![Validate schema component configured with the schema path and content set to the jsonDoc variable](../../assets/blog/json-module-component-in-mule-4-7.png) As per our example, `document.json` does not match `mySchema.json`, so, it throws a `JSON:SCHEMA_NOT_HONOURED` exception. This exception is caught in the error handler and the `error.errorMessage.payload` is written to a file as shown below: ![Write operation in the error handler saving the error payload to errorMessage-pay.csod](../../assets/blog/json-module-component-in-mule-4-8.png) The file will contain the following error message payload: ![JSON validation error output listing schema violations like values too short and missing required properties](../../assets/blog/json-module-component-in-mule-4-9.png) ![Continued JSON validation error report detailing more per-record schema violations](../../assets/blog/json-module-component-in-mule-4-10.png) The JSON Module can throw any of these exceptions: - `JSON:INVALID_INPUT_JSON` - `JSON:INVALID_SCHEMA` - `JSON:SCHEMA_NOT_FOUND` - `JSON:SCHEMA_NOT_HONOURED` Some JSON schemas might reference other schemas through a public URI. However, you might not want to fetch those schemas from the internet, mainly for performance and security reasons. The JSON Module has an option to redirect the schema, which lets you replace an external reference with a local one, without the need to modify the schema itself. The JSON Module also has an option called "Allow Duplicate Keys", which when set true, it will allow the duplication of keys; otherwise, it will fail. This is why the JSON Module is a very useful component for schema validation. Enjoy the benefits of using this component :) Thank you for reading. -Pravallika Nagaraja ## GitHub repository [ProstDev GitHub - JSON Module Component Mule 4](https://github.com/ProstDev/JSONModuleComponentMule4) --- ## Intro to Regression Testing with Postman Source: https://prostdev.com/post/intro-to-regression-testing-with-postman | Published: Jun 8, 2020 | Category: Tutorials *Gist file with the tests script and GitHub repository with the Postman collection can be found at the end of the post.* There are different types of software tests that a team can build to make sure that the final product will work as expected. However, in this post, we’ll focus on one particular type: Regression Testing. ## What is Regression Testing? Regression testing is a type of software testing, in which tests are created to ensure that a new change to the code did not break existing functionality. Let’s illustrate this definition with a metaphor: Imagine that you have a car that works perfectly fine; and you know this, because you checked that everything was working: - Do the lights turn on? Yes. - Does the car start? Yes. - Does the air conditioner work? Yes. - Do the windows go down and up? Yes. - Does the radio work? Yes. - Are my tires pumped up? Yes. Now, let’s say that you installed some neon lights to your car and then when you tried to turn the car on, but guess what? It didn’t work! The lights must’ve done something to the car, because it was working before installing them. Remember the 6 tests that you performed before, to make sure that your entire car was working? Well, now that you installed the neon lights to the car, let’s do these tests once more… - Do the lights turn on? **No**. - Does the car start? **No**. - Does the air conditioner work? **No**. - Do the windows go down and up? **No**. - Does the radio work? **No**. - Are my tires pumped up? Yes. So, a lot of them are failing now, and the ones that are failing are relying on the car’s battery to be able to work properly. It’s possible that when you installed the neon lights, something went wrong and it messed with the battery. You got to this conclusion by paying attention to the failed tests; and you were able to isolate this, because you did one additional test that didn’t have anything to do with the battery, and it did not fail. This set of 6 tests, are actually regression tests. Why? Because you are performing different tests to make sure that the whole car is working properly, after you added new functionality to it. If you add more detailed tests, you will ensure even more that the car doesn’t break with a future change. ## Creating basic tests with Postman You can download Postman for free from their website: [https://www.postman.com/downloads/](https://www.postman.com/downloads/) Once you have it, open it, and it will look something like this: ![Postman opened on the Collections tab with no collections yet created](../../assets/blog/intro-to-regression-testing-with-postman-1.png) On the left side, you have the menu to create a new collection. Collections are basically the folders where you can keep your requests. Let’s create one by clicking on “Create a collection”, and name it however you want. ![A new empty collection named MyNewCollection in the Postman Collections sidebar](../../assets/blog/intro-to-regression-testing-with-postman-2.png) Now, if you click “Add requests”, you can start creating specific calls to your API. I’ll be creating some tests for the Slack API. Here I’m sending a request to `slack.com/api/team.info`, and I’m correctly getting a response back. ![GET to slack.com/api/team.info returning 200 OK with ok:true and team name ProstDev](../../assets/blog/intro-to-regression-testing-with-postman-3.png) I will create some tests for this request, to make sure that the response I get from this call is always successful. To make sure that this is the case, I want to check the following: - That the Content-Type header is present in the response. - That the Content-Type header in the response is of “application/json”. - That there is a response body. - That the response body is in a JSON format. - That the response body contains “ok”: true. - That the team.name in the response is “ProstDev”. - That the response status is 200 OK. I go to the request’s “Tests” tab and I create the previous 7 tests in code (Note that Postman scripts are based on JavaScript), then I simply run the request again and see if the tests passed or failed from the “Test Results” tab: ![Postman Tests tab with seven test scripts and all 7 passing in the Test Results panel](../../assets/blog/intro-to-regression-testing-with-postman-4.png) As you can see, all of them passed as expected. But this is only one request that can be done to the Slack API, if I were to test more, or most of the functionality from it, I would have to add **at least** one request per method, each with their own set of tests. The number of tests and the number of requests depend on the amount of detail that you want to achieve with the testing, or how many different results you can get from each request. The previous request was checking that the response was a "successful" response - this means, no error whatsoever. But now I'm going to create a *negative scenario* - this means that the response that I'm expecting to get is supposed to be an error, because I'm testing that an incorrect request **does** return an error. So, for this incorrect request, I'll add tests that check that I am indeed getting an error back, and that I am actually getting the error that I was expecting to receive. Here’s the response I get when trying to call `slack.com/api/unexistent`: ![Slack response with 200 OK but ok:false and error unknown_method for an invalid method](../../assets/blog/intro-to-regression-testing-with-postman-5.png) This time we received “ok”: false in the response, and an error that specifies “unknown_method”. You will notice, however, that the status code remained as 200 and we didn’t receive any of the other possible error codes (3xx, 4xx, 5xx). This is Slack’s specific design for their API. We will create some tests that check for all these as well. ![Negative-scenario test scripts checking ok:false and error unknown_method, all 7 passing](../../assets/blog/intro-to-regression-testing-with-postman-6.png) We copy/pasted the same tests we had before, but made some minor tweaks to match our response for the negative scenario. We changed test [#5](https://www.prostdev.com/blog/hashtags/5) to search for “ok”: false instead of “ok”: true, and test [#6](https://www.prostdev.com/blog/hashtags/6) to search for “error”: “unknown_method”. Note that we didn’t have to modify test [#7](https://www.prostdev.com/blog/hashtags/7), because the status code is still 200, even though the request returned an error. Now, in real life, you’d have created a lot more requests and tests to make sure that the API would not break with new functionality. Once all your tests are created, you don’t have to run all of them one at a time; you can click on the “Runner” button, located on the top-left side of the app. This will open a new window that looks like this: ![Postman Collection Runner with the Slack-API collection selected before starting a run](../../assets/blog/intro-to-regression-testing-with-postman-7.png) Here you can simply select your collection, and click on the blue “Run” button at the bottom. Postman’s Runner will run all of your requests and tests, and will give you a visual report: ![Collection Runner results: 14 passed, 0 failed across the team.info and negative-test requests](../../assets/blog/intro-to-regression-testing-with-postman-8.png) In this case, all our tests were successful. How would this help me in the future? Let’s imagine that you, as a developer, introduce a new functionality to the Slack API. You’re sure that everything is working, but before continuing with the process of pushing your code to the remote repository, you run the set of Regression Tests, using Postman’s Runner. If even one of the tests fails, you’ll know that you broke some functionality that you weren’t aware of; so, you have to go back to the code and fix that error. After you fix it, you have to run all the tests again to make sure that everything works; and so on, and so on, until all of the tests are passing and your new functionality is working as expected. Let’s do a quick recap of what we learned today: - Regression Testing is one type of software testing, in which tests are created to ensure that a new change to the code did not break existing functionality. - Postman is a great tool to create regression tests for your API, and you can use Postman’s Runner every time that you need to add new features, to make sure that you did not break any existing functionality. Stay tuned for more posts to use Postman as your testing tool! I really recommend it, and you’ll see why ;) *Prost*! -Alex ### GitHub repository [ProstDev GitHub - Postman Collections Examples](https://github.com/ProstDev/postman-collections-examples) ### More links - [HTTP Status Codes](https://www.restapitutorial.com/httpstatuscodes.html) - [Postman](https://www.postman.com/) - [Download Postman](https://www.postman.com/downloads/) - [Slack API](https://api.slack.com/) - [Slack API - team.info method](https://api.slack.com/methods/team.info) Gist file with the first tests script (successful request): ```js var jsonData = pm.response.json(); pm.test("Content-Type is present", function () { pm.response.to.have.header("Content-Type"); }); pm.test("Content-Type is application/json", function () { pm.expect(postman.getResponseHeader("Content-Type")).to.include("application/json"); }); pm.test("Body is present", function () { pm.response.to.have.body(); }); pm.test("Body is a valid json", function () { pm.response.to.be.json; }); pm.test("ok: true", function () { // #5 pm.expect(jsonData.ok).to.eql(true); }); pm.test("team.name: ProstDev", function () { // #6 pm.expect(jsonData.team.name).to.eql("ProstDev"); }); pm.test("Status code is 200", function () { // #7 pm.response.to.have.status(200); }); ``` --- ## Combining Objects: Concatenation in DW 2.0 Source: https://prostdev.com/post/combining-objects-concatenation-in-dw-2-0 | Published: May 28, 2020 | Category: Guides In this post, I will explain what object concatenation is and how to utilize it. You may be familiar with other language’s concatenation functions, especially when using two strings. For example: ``` “Hello”.concat(“ World!”); “Hello” + “ World!”; print(“Hello”, “ World!”) Concatenate ‘string “Hello” “ World!” ``` This can also be achieved in DataWeave with the ++ function. ```dataweave “Hello” ++ “ World” ``` But, can this also be used for objects? Keep reading to find out! ## What is object concatenation? When I say “object concatenation”, I mean the action of combining 2 different objects, and creating one single object containing all the fields (or keys) and values of these objects. For example, we have these 2 JSON objects: ![Two separate JSON objects side by side, one with Key1/Key2 and one with Key4/Key5](../../assets/blog/combining-objects-concatenation-in-dw-2-0-1.png) And we want to achieve this: ![A single merged JSON object containing all six keys from both source objects](../../assets/blog/combining-objects-concatenation-in-dw-2-0-2.png) In the end, we have a single object, that contains everything that the other 2 objects had. This is object concatenation. ## Object concatenation using ++ The most commonly used way to achieve object concatenation in DataWeave, is the ++ function. The previous operation, using DW, would look something like this: ![DataWeave script using object1 ++ object2 and its merged JSON output](../../assets/blog/combining-objects-concatenation-in-dw-2-0-3.png) You can see why this is one of the favourites. It’s easy to read and understand. It is very intuitive as well (object1 plus object2). There is another syntax to do the same thing: ![DataWeave script using prefix ++(object1, object2) syntax producing the same merged output](../../assets/blog/combining-objects-concatenation-in-dw-2-0-4.png) In this syntax, you are writing the function first, and then positioning the first and second arguments inside parentheses, separated by a comma. Either way is correct; it just depends on what you are used to or what is the standard practice in your project. Both parentheses and ++ do the same thing. ## Object concatenation using {( )} This syntax can be cleaner to see in code, but it can also be a bit confusing for new developers if they’re not familiar with the use of {( )}. ![DataWeave script using the parenthesized object-destructor syntax and its merged output](../../assets/blog/combining-objects-concatenation-in-dw-2-0-5.png) I’m sure you’re wondering how this works, if you haven’t seen this syntax before. When there are parentheses ( ) inside curly brackets { }, the parentheses become object destructors. The object in question is separated into pairs of keys and values, and these are then added to the new object that’s being created by the outer curly brackets { }. This transformation would go like this: Step 1: destroy object1 and object2 into their key/value pairs ![Both objects destructured into separate key/value pairs with a blank line between them](../../assets/blog/combining-objects-concatenation-in-dw-2-0-6.png) Step 2: create a new object that contains all these key/value pairs. ![The destructured key/value pairs combined into one new object with all six keys](../../assets/blog/combining-objects-concatenation-in-dw-2-0-7.png) I bet now that you understand how it works, you want to change all your old transformations from using ++ to using {( )}. At least that was my first reaction ;) ## Object concatenation using {( [ ] )} In case you’re not familiar with DataWeave syntax, you can create an array of any type of data, using square brackets [ ]. For example, to create an array of numbers, you’d write it like this: [1, 2, 3, 4, 5]. “Wait, what? Why would I use arrays if I can just use parentheses for my object destructors?” Just hear me out… isn’t it a bit annoying to have to add parentheses for each of the objects? Sometimes I forget to add them and I end up with syntax errors, because you can’t just do this: ![DataWeave script listing two objects without the inner parentheses, raising an invalid input error](../../assets/blog/combining-objects-concatenation-in-dw-2-0-8.png) Turns out that the parentheses destructors not only destroy an object into key/value pairs, they also destroy arrays containing objects! Whaaat? You can do something like this: ![DataWeave script using the {([ object1, object2 ])} array destructor syntax with successful merged output](../../assets/blog/combining-objects-concatenation-in-dw-2-0-9.png) See? Now you can list your objects inside an array (using square brackets [ ]) and then use the object destructor {( )} on that array. Yes, this is the most complex syntax out of the 3 that we saw today. It can be quite confusing for new developers, and you may have to write this explanation out in your internal project’s documentation (if you don’t want to keep repeating the same thing every time you get a new resource). Lucky for you, you can send them the link to this post! In summary, **there are three options** to concatenate objects in DataWeave. 1. You can use the most popular choice by Muleys ```dataweave object1 ++ object2 ++(object1, object2) ``` 2. You can opt for a cleaner, but less known option, using the object destructor: ```dataweave { (object1), (object2) } ``` 3. Or, the most complex one, but without having to surround each object in parentheses: ```dataweave {([ object1, object2 ])} ``` Do you know of any more ways to achieve the same output? Leave me a comment and let me know! Stay safe, get a beer, and *Prost*! -Alex ## More links - [DW Core Functions - PlusPlus](https://docs.mulesoft.com/mule-runtime/4.3/dw-core-functions-plusplus) - [Gist from Alex Martinez - DW 2.0 Cheatsheet](https://gist.github.com/alexandramartinez/f7037eaf970fb02a04017657cd18b8ad) ## Videos - [Getting to the point (under 30 seconds)](/video/3-ways-concatenate-objects-dataweave) - [Complete explanation](https://youtu.be/4OGSdpmtR-s) --- ## MuleSoft Training and Certification Source: https://prostdev.com/post/mulesoft-training-and-certification | Published: May 25, 2017 | Category: Guides Do you sometimes feel like you want to get a certification, but you’re not sure about which one? Or maybe you do, but you have to spend some money just to start the learning process and then pay some more to get the exam. Maybe you start paying for the classes, but then you decide it’s not really what you wanted, so you feel obligated to finish because you already paid for it, but not because it’s what you’d like to do… Some other scenarios come to my mind, but you got the idea. If you ever had some of these problems, or you just want to get a new certification in a new technology, then, you should read this post! ## -MuleSoft? -Yes! MuleSoft! ![Network of connected app and service icons linked by MuleSoft nodes](../../assets/blog/mulesoft-training-and-certification-1.png) Yeah, it sounds really nice, but, what **is** this *MuleSoft?* MuleSoft is the name of the company that created these technologies that help to integrate services. I’ll talk more about what they do and some examples later in this post. About the name, you got it right… Their representative animal is a Mule! But why? > The “mule” in our name comes from the drudgery, or “donkey work,” of data integration that our platform was created to escape. Also, like a mule, we deliver the strength of a donkey to haul the heavy workload, and the speed of a racehorse to get it done quickly. MuleSoft fits the bill. Its name is Max, by the way. Max the Mule. ![Photo grid of Max the Mule figurine posed at travel destinations worldwide](../../assets/blog/mulesoft-training-and-certification-2.png) ## Awesome! But… what is it that they do? They help their clients to achieve a customizable integration with their APIs. It is convenient, because the development can be done without actually touching a line of code and just dragging-and-dropping the components into the MuleSoft Application (using [Anypoint Studio](https://www.mulesoft.com/platform/studio): an IDE based on Eclipse). You can design an API before starting the actual development with [API Designer](https://www.mulesoft.com/platform/api/anypoint-designer), and get the chance to test it before implementing it, using the API Console and the Mocking Service that is integrated with the designer. You can even create tests the very same way you develop the application (with drag-and-drop), using [MUnit](https://www.mulesoft.com/platform/munit-integration-testing). These are just some examples of all the products that MuleSoft offers, you can find the full list in their official website: [www.mulesoft.com](https://www.mulesoft.com/). ## And… what about the trainings? Finally, we get to the coolest part! ![MuleSoft Training and Certification page with Developers, Architects, Operations options](../../assets/blog/mulesoft-training-and-certification-3.png) There is a catalog for different trainings and some of them can get you a certification. Some of them are not free, or have to be taken with an instructor. **BUT!!!** We started this post talking about the costs and the regrets. Hopes and dreams, and stuff… There **is** a 5-days online fundamentals course for you to take whenever it suits you better. It is called [Anypoint Platform Development: Fundamentals](https://training.mulesoft.com/course/development-fundamentals-mule4). The only thing you need to do is to watch the videos and practice with the instructor. You will also receive pdfs with the content. After you finish the training, you will get a voucher in order to take the certification exam for free. Yes. You only have to complete the training, study and practice on your own, and get certified! You have 2 attempts to pass the exam. My advice, practice and read a LOT before taking it. If you don’t make it in the first attempt, don’t worry, you still have another, but take that as a sign and investigate a little bit more about what you didn’t know how to answer. I really hope I could get your attention on a new technology. Even if you don’t get a certification or register to a training, I’m glad you stayed ’til the end of the post and now you have an idea of what MuleSoft is, what it does, and how can you learn. I have to get this out of my chest… I actually completed the training I mentioned before, it was not that easy because of my schedule, but it was really fun, each step of the way. I highly recommend it, even if you don’t want to follow this career path. It opens your mind to a new world (if you’re not familiar with integration technologies), and gives you a newer perspective. I had so much fun using the API Designer, it was the best of the training, in my opinion. Thank you for reading, guys! Get a beer for me, I just got some teeth removals, so I can’t. Don’t forget to say Prost! -Alex --- ## Less time, less cost, less resources, more apps! Source: https://prostdev.com/post/less-time-less-cost-less-resources-more-apps | Published: Aug 23, 2016 | Category: Opinion We’re not supposed to judge a book by its cover, but what happens when you try to get the online menu of your favorite restaurant and you can’t even find their webpage? Or what if you want to get a tattoo, but you can’t find the tattooer’s Facebook page? You may think they’re not professional enough, or they’re not “official”. In today’s world, mobile apps are as important as having an official site for your company, and creating them gets easier and easier. Today we can create a website in minutes and we don’t even have to write a single code line at all! What if you could create a mobile app in some weeks, or months? ## This is Ionic! ![Ionic homepage hero "Create incredible apps" with an iPhone showing an app](../../assets/blog/less-time-less-cost-less-resources-more-apps-1.png) If you’re thinking of creating a new app and you’re worried about choosing Android or iOS, or if you’re a designer and you don’t want to learn how to program an app for two or more different operating systems, then you should check out this framework. You just have to worry about basic HTML, CSS, and Javascript (that, if you want a more personalized app). Besides, you will be learning [Angular](https://angularjs.org/)as you write your new app. > Ionic builds on top of Angular to create a powerful SDK well-suited for building rich and robust mobile apps for the app store and the mobile web. Ionic not only looks nice, but its core architecture is built for serious app development. Once you have a progress on your app and want to try it, you can either run it as a Web App, or use *Cordova* or *Phonegap* for deployment. You can develop one app and get it running on Android or iOS without having to work twice! > Ionic is modeled off of standard native mobile development SDKs, bringing the UI standards of native apps together with the full power and flexibility of the open web. Ionic runs inside [Cordova](http://cordova.apache.org/) or [Phonegap](http://phonegap.com/) to deploy natively, or as a Progressive Web App. Develop once, deploy *everywhere*. But that’s not it. Let me introduce you to [Creator](http://ionic.io/products/creator). ![Ionic Creator landing page "Build amazing mobile apps, faster" with a desk and iMac](../../assets/blog/less-time-less-cost-less-resources-more-apps-2.png) If you’re not into programming, you can always choose to use this tool that Ionic created just for you. You can start a mobile app just by clicking some buttons, and using this friendly *drag and drop* interface. Just click a button for a new project, ![Green "+ New Project" button in the Ionic Creator toolbar](../../assets/blog/less-time-less-cost-less-resources-more-apps-3.png) write the name of your project, choose its type, and one more click to create it. ![New Project dialog naming the app and choosing a type like Side Menu + Tabs](../../assets/blog/less-time-less-cost-less-resources-more-apps-4.png) And now, you have your new app ready to be personalized! ![Ionic Creator drag-and-drop editor with the new app previewed on an iPhone](../../assets/blog/less-time-less-cost-less-resources-more-apps-5.png) You can choose your device, you can see a preview, you can export your app, and more. > Your time is precious. **Creator** has your back. Spend more time on building a great app experience, and less time managing separate code bases and devices. Export cross-platform mobile prototypes that look, feel, and perform amazing across all modern phones and devices. Wether you like code or design, I think this framework is ideal for mobile apps. There is a lot of documentation, if you prefer reading. There are YouTube tutorials, if you learn better that way, or you can just start by yourself! This is just an example of how easy it is to get along with new technologies. If you’re a fan of learning new things, then you will never be bored in this world. New programming languages, new architectures, new tools, new frameworks are being developed right now. One thing is for sure: if you work in this area and you like to learn once and stick with it forever, you will never succeed. Something new comes out, gets popular for some seconds, and then it’s replaced by something newer. I encourage you to continue learning every day, even if it’s just a fun fact you didn’t know before. I hope this post was of your interest, and if you already knew all this, you can contact me and tell me more about it, maybe I don’t know something that you do. Also, if I wrote something wrong, please let me know! I’m human too 🙂 Thank you so much for reading! Now go get a beer and say Prost! -Alex. --- # Claude Code Skills ## accessibility-review Source: https://prostdev.com/skill/accessibility-review | Published: Jul 9, 2026 > [!NOTE] > This one is framework-agnostic on purpose — plain HTML, React, Vue, Svelte, Astro, web components, > it doesn't care. It's the accessibility skill I reach for on any project, not just this site. ## What it does `accessibility-review` is a Claude Code [skill](https://docs.claude.com/en/docs/claude-code/skills) that reviews the accessibility of web UI against **WCAG 2.2, Level A and AA** — the conformance target most teams and regulations actually aim for. You point it at a component, a page, or a diff, and it finds the real violations, tells you which success criterion each one breaks, and gives you the fix. The thing it's careful about: **a finding has to be a clear, direct violation of a specific criterion.** No vague "this could be more accessible" noise that buries the issues that matter. Every reported problem comes with the SC number, the location, the user impact, and a corrected snippet. ## When to use it Ask Claude to review, audit, or fix accessibility on anything that renders a visible or interactive surface — or when the conversation turns to screen readers, keyboard navigation, color contrast, focus management, or [ARIA](https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA) (the accessibility attributes that describe custom widgets to assistive tech). It works while you're *writing* a component too, not just after. > [!NOTE] > **"a11y" = "accessibility."** It's a numeronym — the `a`, the `11` letters in between, and the `y`. > You'll see it everywhere in this space, so it's worth knowing, but this skill spells it out so you > never have to decode it. ## How it works It's a small orchestrator with two reference files it opens on demand (so the skill stays lightweight until it's actually doing a review): 1. **Triage first** — look at what the code renders and match checks to what's actually there. A static paragraph engages almost nothing; a custom `
`-based dropdown engages a dozen criteria. 2. **The checklist** (`reference/wcag-checklist.md`) — ~30 Level A/AA criteria grouped by the four WCAG principles, each with *what it requires → common failures → how to check → the fix with code*, plus an "if the code has X, check Y" triage table. 3. **Verify, don't eyeball** (`reference/verification-guide.md`) — the actual contrast-ratio math (including the trap where `opacity` silently breaks contrast), the keyboard walk, the screen-reader pass, and the zoom/reflow checks. A guessed contrast ratio isn't a finding. Two rules run through the whole thing: **semantic HTML first, ARIA as a last resort** (a native ` Warning… ``` Alt text describes *function* for controls ("Search", not "magnifying glass") and *content/meaning* for images. Keep it concise; no "image of". ### 1.3.1 Info and Relationships — A **Requires:** structure and relationships conveyed visually are also programmatically determinable — via semantic HTML (preferred) or ARIA. Covers lists, tables, form-label association, headings, grouping, and landmark regions. **Common failures:** - **Lists:** visually a list but marked up as `
`s or `
`-separated lines. - **Tables:** data table with no `
`, no `scope`, or a data grid faked with `
`s; missing `
`. - **Form labels:** placeholder used *as* the label; label not associated (`for`/`id` mismatch); a visible label sitting near an input with no programmatic tie. - **Headings:** text that looks like a heading (big/bold) but is a `

`; skipped levels (`

` → `

`); more than one `

` for the page's main title. - **Grouping:** a set of radios/checkboxes with no `
`/``. - **Regions:** whole page in bare `
`s with no `
`/`