grep lines from one file which are in another file

A frequent task in shell scripting and data processing is filtering out lines from one text file that appear in another (e.g., subtracting a list file from a dataset). Rather than writing loops or complex awk scripts, standard GNU grep provides an efficient, one-line solution.

---

The Solution

grep -v -x -F -f list.txt dataset.txt > filtered_output.txt

(This command outputs all lines from dataset.txt that do not match any exact lines listed in list.txt.)

---

Flag Breakdown

Flag	Full Option	Function
-v	--invert-match	Selects non-matching lines (inverts the filter).
-x	--line-regexp	Forces matches to span the entire line, preventing partial substring exclusions.
-F	--fixed-strings	Treats input patterns as literal strings rather than regular expressions (much faster and avoids escaping special characters like ., *, or [).
-f FILE	--file=FILE	Reads exclusion patterns line-by-line from the specified file.

---

Regex Patterns vs. Fixed Strings

• When to omit -F: Use grep -v -x -f patterns.txt data.txt if lines inside patterns.txt contain intentional regular expression syntax (such as wildcard matchers like ^server-[0-9]+).
• When to include -F (Recommended): Use -F whenever you want a strict, exact text match (WYSIWYG). It prevents characters like dots, brackets, and dollar signs from being parsed as regular expressions, while significantly speeding up processing on large datasets.

From Code to Commercials: 10 Harsh Realities Every Engineer Needs to Hear

After more than ten years as an electronics engineer, I have seen brilliant minds get stuck while others thrive in leadership and business. Technical skill is essential, but it is rarely enough on its own. Here are ten direct, practical lessons for building a successful, well-rounded career.

---

1. Plan Your Career Intentionally

Do not drift purely on impulse. Choose a solid industry, gain deep domain knowledge, and avoid frequent job-hopping for minor pay raises. Long-term stability and industry depth build compounding value—once you truly master a sector, financial rewards follow naturally.

---

2. Do Not Get Trapped in Pure Technology

Technical skill is only one pillar of your career value, not the entire foundation. Unless your sole lifetime goal is remaining an individual bench technician, avoid obsessing over narrow technical details at the expense of everything else.

---

3. Cultivate Broad Professional Skills

Never look down on managers or peers with less technical depth. If someone advances, they usually excel at things you might lack: coordinating teams, aligning with leadership priorities, and resolving conflict. Strong communication and diplomacy matter just as much as technical ability.

---

4. Expand Your Social Circle

Do not limit your network strictly to other engineers. Build relationships with people across sales, marketing, finance, and different walks of life. Understanding how different people think and operate is essential if you ever want to lead a business or manage projects.

---

5. Broaden Your Knowledge Base

Depth in your core field is necessary, but breadth makes you versatile. Read and learn about business basics: finance, accounting, taxation, contracts, and logistics. This broad perspective helps you avoid costly mistakes later on.

---

6. Transition to Management or Sales When Ready

Individual technical execution has a natural ceiling. Moving into technical management develops leadership skills, while moving toward sales sharpens commercial intuition and builds an extensive business network—the true drivers of long-term career growth.

---

7. Overcome Classic Engineering Weaknesses

Engineers often struggle with perfectionism, over-analysis, indecision, and thin skin. Overcome these traits through real-world practice: take on public-facing tasks, negotiate with vendors, or participate directly in commercial discussions.

---

8. Build Your Own Workspace and Products

Set up a personal lab at home with essential tools, test equipment, and computing resources. Take on practical side projects to develop market sense. Working products and prototypes carry far more weight with partners and investors than diplomas or certificates.

---

9. Learn to Market Yourself

Skill without visibility leads nowhere. Learn to present your work clearly in writing and speaking. Share knowledge, publish articles, and build a recognizable personal brand so opportunities find you directly.

---

10. Take Action Without Waiting for 100% Certainty

Waiting for complete certainty leads to missed opportunities. When conditions are reasonably good, take decisive action. Success requires practical experimentation and learning through iterative attempts.

Pentaho Data Integration faster setup

If you run Pentaho Data Integration (PDI / Kettle) jobs via kitchen.sh or pan.sh, you may notice that startup latency has steadily increased across modern releases. A basic CLI invocation can take over 10 seconds just to initialize the JVM and load dependencies before processing a single row. By eliminating network reverse-DNS stalls and pruning unused heavy-weight plugins, you can cut CLI startup time down to ~1.2 seconds.

---

Benchmark: Startup Degradation Over Time

Measuring cold startup latency using a minimal test job (time ./kitchen.sh -file=test.kjb):

PDI Release Phase	Real Time (Wall Clock)	User Time (CPU)
Legacy Lightweight Releases	0.49s – 1.10s	0.78s – 2.14s
Early Modular Releases	5.97s – 34.11s	3.86s – 18.86s
Modern Default Installations	11.25s – 12.43s	26.17s – 28.83s
Pruned & Optimized Build	1.25s	3.23s

---

Step 1: Eliminate Hostname & Reverse DNS Lookups

When user CPU time is significantly lower than real clock time, Kettle is idling on network I/O timeouts trying to resolve the local machine's hostname.

• Explicitly define the hostname: Add KETTLE_SYSTEM_HOSTNAME=localhost to your ~/.kettle/kettle.properties file.
• Map the local hostname in /etc/hosts: Ensure your current machine hostname points directly to 127.0.0.1 to prevent remote DNS lookups.
• Force IPv4 Resolution: Pass -Djava.net.preferIPv4Stack=true into your Java options inside kitchen.sh or spoon.sh to bypass IPv6 reverse-lookup timeouts:

  PENTAHO_DI_JAVA_OPTIONS="-Djava.net.preferIPv4Stack=true -Xms512m -Xmx2048m"

---

Step 2: Prune Unused Heavyweight Plugins & OSGi/Karaf Runtimes

Kettle scans and initializes OSGi bundles, Karaf runtimes, Mondrian models, and Big Data shims on every CLI run. If your workflow only uses standard database, text, or transform steps, you can safely remove these subsystems.

Files and Folders to Move Out

Move the following directories, descriptors, and unused engine/driver JARs into a backup location outside your PDI installation directory:

# Move OSGi, Karaf, and Mondrian system runtimes
mv system/karaf /path/to/pdi-backup/
mv system/mondrian /path/to/pdi-backup/
mv system/osgi /path/to/pdi-backup/

# Move heavy plugins
mv plugins/pentaho-big-data-plugin /path/to/pdi-backup/
mv plugins/kettle*-log4j-plugin /path/to/pdi-backup/
mv plugins/pdi-xml-plugin /path/to/pdi-backup/

# Move lifecycle descriptors
mv classes/kettle-lifecycle-listeners.xml /path/to/pdi-backup/
mv classes/kettle-registry-extensions.xml /path/to/pdi-backup/

# Move unused engine, spark, and platform JARs from lib/
mv lib/mondrian-*.jar /path/to/pdi-backup/
mv lib/org.apache.karaf.*.jar /path/to/pdi-backup/
mv lib/pdi-engine-api-*.jar /path/to/pdi-backup/
mv lib/pdi-engine-spark-*.jar /path/to/pdi-backup/
mv lib/pdi-osgi-bridge-core-*.jar /path/to/pdi-backup/
mv lib/pdi-spark-driver-*.jar /path/to/pdi-backup/
mv lib/pentaho-capability-manager-*.jar /path/to/pdi-backup/
mv lib/pentaho-connections-*.jar /path/to/pdi-backup/
mv lib/pentaho-cwm-*.jar /path/to/pdi-backup/
mv lib/pentaho-database-model-*.jar /path/to/pdi-backup/
mv lib/pentaho-hadoop-shims-api-*.jar /path/to/pdi-backup/
mv lib/pentaho-metaverse-api-*.jar /path/to/pdi-backup/
mv lib/pentaho-osgi-utils-api-*.jar /path/to/pdi-backup/
mv lib/pentaho-platform-*.jar /path/to/pdi-backup/
mv lib/pentaho-registry-*.jar /path/to/pdi-backup/
mv lib/pentaho-service-coordinator-*.jar /path/to/pdi-backup/

---

Long-Term Architecture Tip

If you execute hundreds of transformations in rapid batches, repeated JVM cold starts will always introduce cumulative latency. For high-frequency workloads, run transformations via the Carte slave server daemon or integrate directly with the Kettle Java API to keep the JVM warmed up continuously.

group iptables rules in chain

When managing multiple firewall rules for a specific application, service, or policy, adding them directly to default chains (like INPUT, FORWARD, or PREROUTING) makes bulk updates and cleanups difficult. If you need to remove or disable that policy later, you would normally have to delete every single rule one by one.

The cleanest solution in iptables is to create a custom user-defined chain. By grouping related rules inside their own chain and linking that chain to a default table, you can enable, disable, or flush all associated rules in a single command.

---

The Problem: Cluttered Default Chains

Consider applying several packet-marking or filtering rules for a specific service across different hooks:

iptables -t mangle -A PREROUTING -p udp --dport 4444 -j MARK --set-mark 100
iptables -t mangle -A INPUT -p udp --dport 4444 -j MARK --set-mark 100
iptables -t mangle -A OUTPUT -p udp --dport 4444 -j MARK --set-mark 200

Deleting these later requires referencing each specific rule line or rule number individually, which is tedious and error-prone in production environments.

---

The Solution: Group Rules into a Custom Chain

Step 1: Create the Custom Chain in the Target Table

Create a custom chain (e.g., MYCHAIN). Remember that user-defined chains exist only within the specific table where they are created (e.g., mangle, filter, or nat):

# Create MYCHAIN inside the mangle table
iptables -t mangle -N MYCHAIN

Step 2: Add Rules into Your Custom Chain

Populate your chain with the relevant match criteria and actions:

iptables -t mangle -A MYCHAIN -p udp --dport 4444 -j MARK --set-mark 100
iptables -t mangle -A MYCHAIN -p udp --sport 4444 -j MARK --set-mark 200

Step 3: Link Your Custom Chain to Built-in Chains

To make the rules active, direct traffic from the standard hooks (such as PREROUTING or INPUT) into your custom chain:

iptables -t mangle -A PREROUTING -j MYCHAIN
iptables -t mangle -A OUTPUT -j MYCHAIN

---

How to Easily Manage and Delete the Group

When you need to clear or completely remove the rule group, the workflow is clean and immediate:

1. Unlink the Chain: Remove the single jump rule from the built-in chains:

   iptables -t mangle -D PREROUTING -j MYCHAIN
   iptables -t mangle -D OUTPUT -j MYCHAIN

2. Flush the Rules: Clear all rules inside your custom chain in one step:

   iptables -t mangle -F MYCHAIN

3. Delete the Empty Chain: Remove the custom chain definition:

   iptables -t mangle -X MYCHAIN

SSH Remote port mapping

SSH port forwarding (tunneling) is one of the most powerful utilities in a systems administrator's toolkit. It allows you to securely traverse firewalls, access private network services, and expose internal ports without modifying router configurations or setting up full VPNs.

---

Common Flag Breakdown

When creating background tunnels, these flags are commonly combined to keep the session silent, secure, and persistent:

• -f: Requests SSH to go to the background just before command execution (prompts for credentials if needed, then detaches).
• -N: Do not execute a remote command (useful when only forwarding ports).
• -C: Enables gzip compression of all data transferred over the tunnel.
• -g: Allows remote hosts on your local network to connect to local forwarded ports (binds to 0.0.0.0 instead of 127.0.0.1).
• -p <port>: Specifies the remote SSH server daemon port if it differs from the default port 22.

---

1. Local Port Forwarding (-L)

Local forwarding opens a listening port on your client machine and routes incoming traffic through the SSH tunnel to a destination reachable from the remote SSH server.

ssh -C -f -N -g -L [local_bind_port]:[target_destination_ip]:[target_port] user@ssh_jump_host

Practical Scenario: Accessing an internal web dashboard (10.3.32.26:80) located behind an edge gateway (192.168.190.115):

ssh -f -N -l root -L 8500:10.3.32.26:80 192.168.190.115

Visiting http://localhost:8500 in your local browser will now tunnel directly to 10.3.32.26:80.

---

2. Remote (Reverse) Port Forwarding (-R)

Remote forwarding opens a listening port on the remote SSH server and directs all incoming connections back to a designated port accessible from your local system. This allows you to expose a local service to an external server.

ssh -C -f -N -g -R [remote_listen_port]:[target_destination_ip]:[target_port] user@remote_server

Practical Scenario: Forwarding traffic received on remote server 174.139.9.66:8080 directly into your local machine's HTTP service on port 80:

ssh -C -f -N -g -R 8080:127.0.0.1:80 master@174.139.9.66

(Note: For remote clients to connect to the remote port, ensure GatewayPorts yes is set in the remote server's /etc/ssh/sshd_config.)

---

3. Dynamic Application Forwarding / SOCKS5 Proxy (-D)

Instead of mapping a single port, dynamic forwarding allocates a local port acting as a SOCKS4/SOCKS5 proxy. Traffic sent through this proxy is automatically routed dynamically depending on the protocol and requested destination host.

ssh -C -f -N -D 1080 user@proxy_jump_host

Configure your web browser or system network settings to use SOCKS proxy 127.0.0.1:1080 to route all browsing traffic through the remote host securely.

---

Comparison Summary

Forwarding Mode	Flag	Where the Listening Port Lives	Primary Use Case
Local	-L	Client / Local Host	Accessing remote private servers or internal DBs
Remote (Reverse)	-R	Remote SSH Server	Exposing local development services to the internet
Dynamic	-D	Client / Local Host	Full-traffic SOCKS5 proxy via remote jump host