fedallbest/joplin-notes
Watch
1
0
Fork
You've already forked joplin-notes
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

initial

Browse source
This commit is contained in:
nm1 2026-11-04 00:33:23 +05:30
commit 74ae93a96e
23 changed files with 94691 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

87
joplin notion extension/Web Development/nodejs/Basics.md Normal file
View file

@ -0,0 +1,87 @@
Basics
- express static public
```js
const express = require("express");
const fetch = require("node-fetch");
const cors = require("cors");
const PORT=process.env.PORT || 8080;
const app = express();
app.use(express.static(__dirname + "/public"));
app.use(express.json());
const corsOptions = {
origin: 'http://example.com',
credentials: true,
};
app.use(cors(corsOptions));
const server = app.listen(PORT, () => {
console.log(`app is running on port ${PORT}`);
});
```
- asynchronous vs (multithreading, single threaded)
So when an application is single threaded, it can only consume one logical cpu execution context at a time (one thread)
This means the logic happens step-by-step (can only be interrupted by the cpu scheduler)
 
When an app is multithreaded, it creates child processes via forking or process ownership that it communicates to via shared memory or messaging. Whether the multiple threads perform the same work is up to the design of the application
 
asynchronous single threaded : It can perform multiple tasks, not at the same time though.
Node's async functionality just lets blocks of code wait for preconditions before it runs -- it's not truly async with other code
 
for simultaneoulsy request sometimes
In NodeJS, a single instance can use one thread, unless it forks or spawns a worker.
Most node webservers use forking, where they effectively spawn one process per logical thread on the host server. They randomly get traffic by a special "port sharing" feature of Node itself -- which is managed by the cluster module IIRC
- express get with query
```js
app.get("/getaudio", async (req, res) => {
const queryObj = req.query;
console.log("queryobj: ", queryObj);
const { url, title, q } = queryObj;
}
}
```
- express get with params
```js
app.get('/books/:id', (req,res)=>{
const {id}=req.params
} )
```
- express res object
- headers
`res.setHeaders` - calls the native Node.js method
`res.set` - sets headers
`res.headers` - an alias to res.set
```js
res.set('Content-Type', 'text/plain');
res.set({
'Content-Type': 'text/plain',
'Content-Length': '123',
'ETag': '12345'
})
res.header(field, [value])
res.setHeader(
"Access-Control-Allow-Methods",
"GET,OPTIONS,PATCH,DELETE,POST,PUT"
);
```
- middleware
https://www.geeksforgeeks.org/middleware-in-express-js/
 
to only install production dependencies use
`npm install --omit=dev`
for only dev dependenciues
`npm install --only=dev`
for both
`npm i`

33
joplin notion extension/Web Development/nodejs/Downloading and transferring blobs.md Normal file
View file

@ -0,0 +1,33 @@
Downloading and transferring blobs
```js
fetch(url)
.then((res) => res.blob())
.then(async (blob) => {
console.log("got blob");
console.log("Took " + (Date.now() - initialTime) / 1000 + "s");
// const title = "video";
// const file = await res.blob();
const arrayBuffer = await blob.arrayBuffer();
const buffer = Buffer.from(arrayBuffer);
// console.log("bytelength: ", buffer.byteLength());
// const arrayByte = new Uint8Array(arrayBuffer);
res.writeHead(200, {
"content-type": "application/octet-stream",
});
console.log("converted to buffer");
res.write(buffer);
// const writeStream = fs.createWriteStream(arrayByte);
// res.sendFile(buffer);
res.end();
// const buffer = Buffer.from(file, "binary");
// file.pipe(fs.createWriteStream(`${title}.mp4`));
// fs.writeFileSync(`${title}12.mp4`, buffer);
})
.catch((err) => console.log(err));
```

175
joplin notion extension/Web Development/nodejs/Extras.md Normal file
View file

@ -0,0 +1,175 @@
Extras
- https://stackoverflow.com/questions/36022926/what-do-the-save-flags-do-with-npm-install
`npm i <package_name> --save` and `npm i <package_name> ` are same now and they also update dependencies of package too. to not update dependencies of package use `npm i <package_name> --no-save`
- nvm(Node version manager)
`curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash`
 
`nvm ls-remote` : for getting list of available versions which can be downloaded
`nvm list` : for getting list of downloaded versions of node
`nvm install 14.0.0` : for downloading and installing version 14.0.0
`nvm use 14.0.0` : switching to version 14.0.0
`nvm use system`: switching to version when nodejs first installed
`nvm alias default 18.0`: this will make 18 as default
- https://github.com/ishtms/learn-nodejs-hard-way (without installing other frameworks)
- https://youtu.be/Nh-252u80F4 : get wifi password
- Running nodejs in background
https://sentrytwo.com/92df06609b
- To avoid running sudo
Why You Should Never Run "sudo npm"... https://www.youtube.com/watch?v=aFrR56sqC_A
- **pm2**
PM2 is a production process manager for Node.js applications with a built-in load balancer. It allows you to keep applications alive forever, to reload them without downtime and to facilitate common system admin tasks.
install this npm package globally or locally.
`npm i -g pm2`
Using this we can start multiple node instances with single command. But first starting node application instances in background
`pm2 start app.js --name app1` same as `node app.js` but in background
to stop node instance use `pm2 stop app1`
multiple node instance in file `ecosystem.config.js`
```js
module.exports = {
apps: [
{
name: 'main',
script: 'index.js',
watch: true,
env: {
NODE_ENV: 'production',
PORT: 8081
}
},
{
name: 'ytdownloadspace',
script: '../ytDownloadSpace/app.js',
watch: true,
env: {
NODE_ENV: 'production',
PORT: 8080
}
},
{
name: 'ytdlApi',
script: '../ytdlApi/app.js',
watch: true,
env: {
NODE_ENV: 'production',
PORT: 8083
}
}
]
};
```
and fiannly when use
`pm2 start ecosystem-config.js`.
To stop all these started with this file use
`pm2 stop ecosystem-config.js`
To list processes use
`pm2 list`
- installing packages through repo link
`npm i https://github.com/expressjs/express`
https://www.pluralsight.com/resources/blog/guides/install-npm-packages-from-gitgithub
&nbsp;
- download a file with https and fs module
```js
const https = require("https"); // Use 'http' for non-HTTPS links
const fs = require("fs");
const fileUrl =
"https://cdn.discordapp.com/attachments/1473231612341915668/1473231989501853819/packages_sorted2.txt?ex=699575b5&is=69942435&hm=66884969a27901525409ff1e5400619ea5305667a130476b7982e041c55297e8&"; // Replace with your file URL
const destinationPath = "./downloaded-file.txt"; // Replace with desired local path and filename
const file = fs.createWriteStream(destinationPath);
https
.get(fileUrl, (response) => {
// Check if the request was successful (status code 200)
if (response.statusCode !== 200) {
console.error(`Download failed, status code: ${response.statusCode}`);
file.close(); // Close the write stream
fs.unlink(destinationPath, () => {}); // Delete the partial file
return;
}
// Pipe the response stream to the file stream
response.pipe(file);
// Event listener for when the download is complete
file.on("finish", () => {
file.close();
console.log("Download Completed!");
});
})
.on("error", (err) => {
// Handle network errors
fs.unlink(destinationPath, () => {}); // Delete the partial file on error
console.error("Error during download:", err.message);
});
```
&nbsp;
tips
1. to prevent adding of installed packages in package.json use `--no-save` flag when installing packages. eg: `npm i --no-save sqlite3`
&nbsp;
&nbsp;
better-sqlite guide
https://www.w3resource.com/sqlite/snippets/better-sqlite3.php
[Comprehensive Guide to Better-SQLite3 (16_5_2026 5：48：16 am).html](../../../_resources/Comprehensive%20Guide%20to%20Better-SQLite3%20%2816_5_2026%205.html)
for parameterized query, string concatenation using `||`
```js
selectQuery = `select * from ${songDataTableEnums.tablename}
where ${songDataTableEnums.columns.songname} like ? || '%' or ${songDataTableEnums.columns.songname} like '% '|| ? || '%'
or ${songDataTableEnums.columns.artist_name} like ? || '%' or ${songDataTableEnums.columns.artist_name} like '% '|| ? || '%'`;
```
&nbsp;
generate thumbnails of video
```js
const ffmpeg = require('fluent-ffmpeg');
ffmpeg('path/to/video.mp4')
.screenshots({
count: 1,
timemarks: ['00:00:02'], // Capture at 2 seconds
filename: 'thumbnail.png',
folder: 'output/path/'
});
```
or
```js
ffmpeg('input.mp4')
.screenshots({
count: 3, // Number of screenshots to take
timestamps: ['10%', '30sec', 60], // Locations: percentages, strings, or numbers (seconds)
filename: 'thumbnail-at-%s-seconds.png', // %s inserts the timestamp
folder: './output', // Destination directory
size: '320x240' // Rescales output width x height
})
.on('end', () => {
console.log('Screenshots taken successfully!');
})
.on('error', (err) => {
console.error('An error occurred: ' + err.message);
});
```
* count: Total number of images to generate. If timestamps is not provided, it automatically distributes screenshots evenly across the video duration.
* timestamps: An array specifying exact positions to snapshot.
* Numbers represent absolute seconds (e.g., 15).
* Strings format time as HH:MM:SS.milli or specific counts (e.g., '00:01:20').
* Percentages evaluate relative to video length (e.g., '50%').
* folder: Path to the directory where files will save.
* filename: Format for file naming. Use %s to interpolate the specific frame's timestamp or %i for the index sequence.

32
joplin notion extension/Web Development/nodejs/HTML parser.md Normal file
View file

@ -0,0 +1,32 @@
HTML parser
- jsdom
It's an npm library that can take html as input and let you do things like document.querySelector on it inside of NodeJS without running a sandboxed browser instance
- node-html-parser : like jsdom
https://www.npmjs.com/package/node-html-parser
eg
```js
import { parse } from 'node-html-parser';
const root = parse('<ul id="list"><li>Hello World</li></ul>');
console.log(root.firstChild.structure);
// ul#list
// li
// #text
console.log(root.querySelector('#list'));
// { tagName: 'ul',
// rawAttrs: 'id="list"',
// childNodes:
// [ { tagName: 'li',
// rawAttrs: '',
// childNodes: [Object],
// classNames: [] } ],
// id: 'list',
// classNames: [] }
console.log(root.toString());
// <ul id="list"><li>Hello World</li></ul>
root.set_content('<li>Hello World</li>');
root.toString(); // <li>Hello World</li>
```

4
joplin notion extension/Web Development/nodejs/Issues.md Normal file
View file

@ -0,0 +1,4 @@
- To clean cache
`npm cache clean --force`
`npm cache verify`

19
joplin notion extension/Web Development/nodejs/bash commands in nodejs.md Normal file
View file

@ -0,0 +1,19 @@
bash commands in nodejs
```js
const { exec } = require('child_process');
const util = require('util');
const execPromise = util.promisify(exec);
async function runCommand() {
try {
const { stdout, stderr } = await execPromise('echo "Hello from Bash!"');
console.log(stdout);
} catch (error) {
console.error(`Error: ${error.message}`);
}
}
runCommand();
```

30
joplin notion extension/Web Development/nodejs/cloudfare.md Normal file
View file

@ -0,0 +1,30 @@
cloudfare
- using it for tunnel
https://yewtu.be/watch?v=Dp5VcCmlKh4
`cloudfared tunnel --url http://localhost:8080`
- deploy nodejs app
> it doesn't uses nodejs engine
<https://developers.cloudflare.com/workers/runtime-apis/nodejs/>
> express module isn't supported
<https://hono.dev/> or <https://github.com/kwhitley/itty-router>
> it don't give access to file system
<https://www.cloudflare.com/developer-platform/r2/>
> It's for basic things and simple requests
No, you just have to build your app a bit differently
- Example
- cors-anywhere
https://github.com/Zibri/cloudflare-cors-anywhere
[cloudflare-cors-anywhere-master.zip](../../../_resources/cloudflare-cors-anywhere-master.zip)
- cloudflare workers
Use wrangler by installing it via npm then
login with
`wrangler login`
if worker name is test then
`wrangler init --from-dash test`
Then finally deploy with
`wrangler deploy` or `wrangler publish`

40
joplin notion extension/Web Development/nodejs/curl relation.md Normal file
View file

@ -0,0 +1,40 @@
curl relation
- `curl -F "reqtype=fileupload" -F "userhash=####" -F "fileToUpload=@cutie.png" https://catbox.moe/user/api.php`
Converting to node-fetch
```js
import fetch, { FormData, fileFromSync } from 'node-fetch';
const form = new FormData();
form.append('reqtype', 'fileupload');
form.append('userhash', '####');
form.append('fileToUpload', fileFromSync('cutie.png'));
fetch('https://catbox.moe/user/api.php', {
method: 'POST',
body: form
});
```
Converting to axios
```js
import axios from 'axios';
import FormData from 'form-data';
import * as fs from 'fs';
const form = new FormData();
form.append('reqtype', 'fileupload');
form.append('userhash', '####');
form.append('fileToUpload', fs.readFileSync('cutie.png'), 'cutie.png');
const response = await axios.post(
'https://catbox.moe/user/api.php',
form,
{
headers: {
...form.getHeaders()
}
}
);
```

61
joplin notion extension/Web Development/nodejs/ejs.md Normal file
View file

@ -0,0 +1,61 @@
ejs
https://www.digitalocean.com/community/tutorials/how-to-use-ejs-to-template-your-node-application
![How To Use EJS to Template Your Node Application _ DigitalOcean.pdf](../../../_resources/How%20To%20Use%20EJS%20to%20Template%20Your%20Node%20Application%20_.pdf)
layouts
use this for layouts https://www.npmjs.com/package/express-ejs-layouts
```js
const express = require('express');
const expressLayouts = require('express-ejs-layouts');
const app = express();
// Set EJS as the view engine
app.set('view engine', 'ejs');
// Use the layouts middleware
app.use(expressLayouts);
// Optional: Define a default layout file
app.set('layout', 'layouts/layout');
```
layout at `views/layouts/layout.ejs`
```html
...
<body>
<main>
<!-- Individual page content is injected here -->
<%- body %>
</main>
</body>
```
view at `views/index.ejs`
```html
<h1>Welcome to my Homepage!</h1>
<p>This is the main content of the home page.</p>
```
for custom layout
```js
app.get('/admin', (req, res) => {
res.render('admin-dashboard', { layout: 'layouts/admin-layout' });
});
```
&nbsp;
when using variable which might be undefined this is safest way to do it
```html
<% if( typeof keyword!=='undefined' && keyword ){ %>
<br>
<% } %>
```

470
joplin notion extension/Web Development/nodejs/file split and merge (express and multer).md Normal file
View file

@ -0,0 +1,470 @@
file split and merge (express and multer)
frontEnd
```html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Document</title>
</head>
<body>
<input id="fileInput" type="file" name="" id="" multiple ><br>
<button onclick="submit()" id="submit">Submit</button>
<script src="script.js"></script>
</body>
</html>
```
js
```jsx
const fileEle=document.querySelector("#fileInput")
const sliceSize=10*1024*1024 // 10MB
const promiseSetTimeOut = (time) => {
return new Promise((resolve) => {
setTimeout(() => {
resolve();
}, time);
});
};
const submitIndividual=async(file,index,total)=>{
const fileSize=file.size
const fileName=file.name
console.log('file: ',file)
let start=0;
let end=sliceSize
const filePieces=[]
let i=0;
while(end<=fileSize)
{
const chunk = file.slice(start,end)
if(end==fileSize)
{
filePieces.push({pos: i,chunk,isEnd: true,fileName})
}
else{
filePieces.push({pos: i,chunk,isEnd: false})
}
start=end
end+=sliceSize
i++
}
if(start<fileSize)
{
const chunk = file.slice(start,fileSize)
filePieces.push({pos: i,chunk,isEnd: true,fileName})
i++
}
for(let filePiece of filePieces )
{
const form=new FormData()
// form.append('data',{pos: filePiece.pos,isEnd: filePiece.isEnd} )
form.append('pos',filePiece.pos )
form.append('isEnd',filePiece.isEnd)
form.append('file',filePiece.chunk)
form.append('fileName',filePiece.fileName)
form.append('total',total)
form.append('fileIndex',index)
const res=await fetch('/submit',{
method: 'post',
// headers: {
// 'Content-Type': 'multipart/form-data'
// },
body: form
})
const data=await res.json()
await promiseSetTimeOut(2000)
}
console.log('file: ',file)
console.log('filePieces: ',filePieces)
}
const submit=async()=>{
for(let file of fileEle.files)
{
await submitIndividual(file,i,fileEle.files.length)
}
// const blob=new Blob(filePieces)
// const a=document.createElement("a")
// const link=URL.createObjectURL(blob)
// a.href= link
// a.download="something1"
// a.click()
}
```
&nbsp;
Nodejs
```js
const express=require('express')
const fs=require('fs')
const FormData= require('form-data')
const app=express()
const PORT= process.env.PORT||8080
const multer=require('multer')
const upload = multer({ dest: 'uploads/' });
// Define the fields you want to handle
const fieldsToParse = [
{ name: 'file', maxCount: 1 },
{ name: 'pos', maxCount: 1 },
{ name: 'isEnd', maxCount: 1 },
{ name: 'fileName', maxCount: 1 },
{ name: 'total', maxCount: 1 },
{ name: 'fileIndex', maxCount: 1 },
];
app.use(express.static(__dirname+'/front'))
app.use(express.json());
app.use(express.urlencoded({extended: true}) )
// 2d array. outer dimension containing files and inner dimension containing file pieces
const fileObjs=[]
let total=0
const getLast=(arr)=>{
if(arr.length>0)
return arr[arr.length-1]
else{
return null
}
}
const mergeFile=()=>{
// mergeFile will happen when only last filepiece is recieved so now
// it will merge
const filePieces=getLast(fileObjs)
const arrBuffer=[]
let ogname=""
for(let filePiece of filePieces)
{
const pieceData=fs.readFileSync(`uploads/${filePiece.name}`)
fs.unlinkSync(`uploads/${filePiece.name}`)
arrBuffer.push(pieceData)
ogname=filePiece.ogName
}
console.log('arrBuffer: ',arrBuffer)
const mergedFileBuffer=Buffer.concat(arrBuffer)
fs.writeFileSync(`uploads/${ogname}`,mergedFileBuffer)
}
app.post('/submit',upload.fields(fieldsToParse),(req,res)=>{
try{
if(total==0)
{
fileLinks.splice(0, fileLinks.length);
}
const file = req.files['file'][0]; // Access the uploaded file
const pos = req.body['pos']; // Access 'pos' field
const isEnd = req.body['isEnd'];
const fileName=req.body['fileName']
total= Number(req.body['total'])
const fileIndex=Number(req.body['fileIndex'])
// determining if current piece is beginning or not
let isStart=false
// const body=req.body
// const file=req.file
console.log('body: ',req.body)
// console.log('body.data: ',.data)
// console.log('body.pos: ',body.pos)
console.log('file: ',file)
// console.log('pos: ',pos)
// console.log('isEnd: ',isEnd)
// console.log('fileName: ',fileName)
const filePieceObj={
pos,
isEnd,
file,
ogName: fileName,
name: file.filename
}
// if array is empty or last of piece of last file 'isEnd' is true
// last piece of last file bcz so that next time when file batch comes
// then for next pieces of first file 'isStart': true
if(fileObjs.length==0 || getLast(getLast(fileObjs)).isEnd==='true' )
{
isStart=true
}
else{
isStart=false
}
if(isStart)
{
fileObjs.push([filePieceObj])
}
else{
getLast(fileObjs).push(filePieceObj)
}
if(filePieceObj.isEnd==='true')
{
console.log('isEnd true ')
mergeFile()
// when all files are recieved here
if(fileIndex===total-1)
{
console.log('final end reached')
message+=" All files uploaded."
total=0
res.json({status: 'ok',message, isCompleted: true})
return
}
}
console.log('fileObjs: ',fileObjs)
res.json({ status: 'ok',message: 'uploaded successfully'})
}
catch(err){
console.log('err: ',err)
res.json({status: 'error'})
}
})
const server=app.listen(PORT,()=>{
console.log('app is running in PORT ',PORT)
})
```
&nbsp;
Generalized version
Front end
```js
// submitIndividualGeneral(file,HOST/cors-submit/chunks?url=<encoded_url> ,resType,bodyType,method,bodyToBeSent,fileFieldName,chunksize)
// after fetch when link is ready message is filelink is expected and status and message field are optional
// bodyType is either 'form' or 'file'
// method is either 'post' or 'put'
const submitIndividualGeneral = async (
file,
url,
resType = "json",
bodyType = "form",
method = "post",
bodyToBeSent = {},
fileFieldName = "",
chunksize = sliceSize
) => {
const fileSize = file.size;
const fileName = file.name;
try {
console.log("file: ", file);
let start = 0;
let end = chunksize;
const filePieces = [];
let i = 0;
while (end <= fileSize) {
const chunk = file.slice(start, end);
if (end == fileSize) {
filePieces.push({ pos: i, chunk, isEnd: true, fileName });
} else {
filePieces.push({ pos: i, chunk, isEnd: false });
}
start = end;
end += chunksize;
i++;
}
if (start < fileSize) {
const chunk = file.slice(start, fileSize);
filePieces.push({ pos: i, chunk, isEnd: true, fileName });
i++;
}
// let fileLinks = [];
for (let filePiece of filePieces) {
const form = new FormData();
// form.append('data',{pos: filePiece.pos,isEnd: filePiece.isEnd} )
let res;
form.append("pos", filePiece.pos);
form.append("isEnd", filePiece.isEnd);
form.append("file", filePiece.chunk);
form.append("fileName", filePiece.fileName);
form.append("bodyToBeSent", JSON.stringify(bodyToBeSent));
// body to be sent to proxy will be form but from proxy to target will be file if bodytype is file
form.append("bodyType", bodyType);
if (bodyType === "form") {
form.append("fileFieldName", fileFieldName);
} else {
form.append("fileFieldName", "fname");
}
// form.append("total", total);
// form.append("fileIndex", index);
res = await fetch(url, {
method,
// headers: {
// 'Content-Type': 'multipart/form-data'
// },
body: form,
});
let data;
if (resType === "json") {
data = await res.json();
} else if (resType === "text") {
data = await res.text();
}
console.log("res: ", data);
if (filePiece.isEnd === true) {
// const fileLink = data;
return { status: "ok", data };
}
await promiseSetTimeOut(2000);
}
console.log("file: ", file);
console.log("filePieces: ", filePieces);
return { status: "error" };
} catch (err) {
console.log("err: ", err);
// return { status: "error", message: err };
throw err;
}
};
const encodedURL = encodeURIComponent(apiUrl);
// const proxyURL='http://localhost:8080/cors-submit'
// const proxyURL = `${CORS_PROXY}/cors-submit`;
const proxyURL = `${CORS_PROXY}/cors-submit/chunks`;
const fetchurl = `${proxyURL}?url=${encodedURL}`;
const resData = await submitIndividualGeneral(
file,
fetchurl,
"text",
"file",
"put"
);
```
&nbsp;
### splitting in nodejs instead of browser
using buffers
```js
const fs = require('fs');
const fileBuffer = fs.readFileSync('small-file.zip'); // Entire file loaded into RAM
const CHUNK_SIZE = 50 * 1024 * 1024;
for (let i = 0; i < fileBuffer.length; i += CHUNK_SIZE) {
const chunk = fileBuffer.slice(i, i + CHUNK_SIZE);
fs.writeFileSync(`part-${i / CHUNK_SIZE}.part`, chunk);
}
```
using streams for efficiency
```js
const fs = require('fs');
const path = require('path');
// Configuration: 50MB in bytes
const CHUNK_SIZE = 50 * 1024 * 1024;
const sourceFile = 'your-large-file.zip';
function splitFile(filePath) {
const stats = fs.statSync(filePath);
const totalParts = Math.ceil(stats.size / CHUNK_SIZE);
const fileName = path.basename(filePath);
for (let i = 0; i < totalParts; i++) {
const start = i * CHUNK_SIZE;
// The 'end' is inclusive, so we subtract 1
const end = Math.min((i + 1) * CHUNK_SIZE - 1, stats.size - 1);
const readStream = fs.createReadStream(filePath, { start, end });
const partName = `${fileName}.part${(i + 1).toString().padStart(3, '0')}`;
const writeStream = fs.createWriteStream(partName);
readStream.pipe(writeStream);
readStream.on('end', () => {
console.log(`Finished writing ${partName}`);
});
}
}
splitFile(sourceFile);
```

106
joplin notion extension/Web Development/nodejs/file upload as body directly.md Normal file
View file

@ -0,0 +1,106 @@
file upload as body directly
We need to pipe the data to writestream for this
```js
const express = require("express");
const fs = require("fs");
const app = express();
const port = 8090;
app.use(express.json({ limit: "10mb" })); // Adjust the limit as needed
app.use(express.static(__dirname + "/public"));
// Handle PUT request with file in the request body
app.put("/upload", (req, res) => {
console.log("body: ", req.body);
console.log("file: ", req.files);
const wsStream = fs.createWriteStream("uploads/name.mp3");
req.pipe(wsStream);
req.on("end", () => {
wsStream.close(() => {
res.json({ data: "success" });
});
});
});
app.listen(port, () => {
console.log(`Server is running on port ${port}`);
});
```
Front end
```html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Document</title>
</head>
<body>
<input type="file" name="" value="">
<button>Submit</button>
<script>
const convertToFormData=(obj)=>{
const formData=new FormData()
for(let prop in obj){
if (obj[prop] instanceof File) {
formData.append(prop, obj[prop], obj[prop].name);
}
else {
formData.append(prop, obj[prop]);
}
}
return formData
}
const fileEle=document.querySelector("input")
const button=document.querySelector("button")
button.addEventListener("click",()=>{
// const url="http://localhost:8085/post"
// const encodedURL=encodeURIComponent(url)
// const proxyURL='http://localhost:8080/cors-submit'
// const fetchurl=`${proxyURL}?url=${encodedURL}`
const fetchurl='upload'
const fileInput=fileEle.files[0]
const obj={
username: 'hey',
pass: '123',
fileInput
}
fetch(fetchurl,{
method : 'PUT',
mode: 'cors',
headers: {
//'Content-Type' : "multipart/form-data",
},
body : fileInput
})
.then(res=>res.json())
.then(data=>{
console.log('data: ',data)
})
.catch(err=>{
console.log('err: ',err)
})
} )
</script>
</body>
</html>
```

17
joplin notion extension/Web Development/nodejs/logging in text file.md Normal file
View file

@ -0,0 +1,17 @@
logging in text file
in order for console.log to write output in log.txt file
```js
const consoleWrite = (...data) => {
const date = new Date();
// date.toLocaleString()
fs.appendFileSync(`${__dirname}/log.txt`, `${date.toLocaleString()}\n`);
fs.appendFileSync(`${__dirname}/log.txt`, JSON.stringify(data));
fs.appendFileSync(`${__dirname}/log.txt`, "\n\n");
};
console.log = consoleWrite;
```

176
joplin notion extension/Web Development/nodejs/multer.md Normal file
View file

@ -0,0 +1,176 @@
multer
https://blog.logrocket.com/multer-nodejs-express-upload-file/
![Multer Easily upload files with Node.js and Express - LogRocket Blog.pdf](../../../_resources/Multer%20Easily%20upload%20files%20with%20Node.js%20and%20Expres.pdf)
- By default express dont support files in formdata so we need this module
```js
const multer = require('multer');
const app = express();
// Create a multer instance and specify destination for uploaded files
const upload = multer({ dest: 'uploads/' });
// Handle form data and file uploads using multer middleware
app.post('/submit-form', upload.single('fileInput'), (req, res) => {
// Access form data from req.body
const formData = req.body;
// Access uploaded file details from req.file
const uploadedFile = req.file;
// Process the form data and uploaded file
// ...
// Send a response
res.send('File uploaded and form data received.');
});
// Start the server
const PORT = 3000;
app.listen(PORT, () => {
console.log(`Server is running on port ${PORT}`);
});
```
req.file is js object like this
```js
{
fieldname: 'file', // Name of the form field specified in HTML
originalname: 'photo.jpg', // Name of the file on the user's computer
encoding: '7bit', // Encoding type of the file
mimetype: 'image/jpeg', // The MIME type of the file
destination: 'uploads/', // The folder where the file has been saved
filename: '1714632000-photo.jpg', // The name of the file within the destination
path: 'uploads/1714632000-photo.jpg', // The full path to the uploaded file
size: 354612 // Size of the file in bytes
}
```
- For multiple file uploads
```js
app.post('/submit-form', upload.array('fileInputs', 3), (req, res) => {
// Access form data from req.body
const formData = req.body;
// Access uploaded files details from req.files
const uploadedFiles = req.files;
```
- For dynamic file fieldname
```js
app.post('/submit-form', upload.any(), (req, res) => {
// Access form data from req.body
const formData = req.body;
// Access uploaded files details from req.files
const uploadedFiles = req.files;
```
Here 2nd argument in upload.array() ie 3 is optional. fileInputs is required name which is name of field containing files
&nbsp;
- Frontend part
```html
<body>
<div class="container">
<h1>File Upload</h1>
<form id='form'>
<div class="input-group">
<label for='name'>Your name</label>
<input name='name' id='name' placeholder="Enter your name" />
</div>
<div class="input-group">
<label for='files'>Select files</label>
<input id='files' type="file" multiple>
</div>
<button class="submit-btn" type='submit'>Upload</button>
</form>
</div>
<script src='./script.js'></script>
</body>
```
&nbsp;
```css
/* style.css */
body {
background-color: rgb(6, 26, 27);
}
* {
box-sizing: border-box;
}
.container {
max-width: 500px;
margin: 60px auto;
}
.container h1 {
text-align: center;
color: white;
}
form {
background-color: white;
padding: 30px;
}
form .input-group {
margin-bottom: 15px;
}
form label {
display: block;
margin-bottom: 10px;
}
form input {
padding: 12px 20px;
width: 100%;
border: 1px solid #ccc;
}
.submit-btn {
width: 100%;
border: none;
background: rgb(37, 83, 3);
font-size: 18px;
color: white;
border-radius: 3px;
padding: 20px;
text-align: center;
}
```
```js
// script.js
const form = document.getElementById("form");
form.addEventListener("submit", submitForm);
function submitForm(e) {
e.preventDefault();
const name = document.getElementById("name");
const files = document.getElementById("files");
const formData = new FormData();
formData.append("name", name.value);
for(let i =0; i < files.files.length; i++) {
formData.append("files", files.files[i]);
}
fetch("http://localhost:5000/upload_files", {
method: 'POST',
body: formData,
headers: {
"Content-Type": "multipart/form-data"
}
})
.then((res) => console.log(res))
.catch((err) => ("Error occured", err));
}
```
or using just javascript instead of form element in html
```js
const files = document.getElementById("files").files;
const formData = new FormData();
files.forEach((file) => {
formData.append("fileInputs", file); // Append files to the FormData object
});
// this header
// "content-type" : "multipart/form-data"
// can cause error so if that occurs dont include this header
```

97
joplin notion extension/Web Development/nodejs/mysql connect.md Normal file
View file

@ -0,0 +1,97 @@
mysql connect
Install it first using `npm i mysql2` as mysql v1 may have some issues
```js
const mysql = require("mysql");
const con = mysql.createConnection({
host: "localhost",
user: "root",
password: "password",
database: "hospitals",
multipleStatements: true,
});
app.post("/register", (req, res) => {
const { name, address, email, phone_no, password } = req.body;
const { patient_photo } = req.files;
const patientPhotoPathname = patient_photo.path;
// Store the registration details
con.query(
`use hospitals;
insert into patients(name,address,email,phone_no,password,patient_photo_path)
values ('${name}','${address}','${email}','${phone_no}','${password}','${patientPhotoPathname}')
`,
(err, result) => {
if (err) {
console.log(err);
res
.status(400)
.json("Error while registering. Email address must be unique");
} else {
console.log(result);
res.json("Registered");
}
}
);
}
app.get("/fetch", (req, res) => {
const { hospital_id } = req.body;
// console.log("body:", req.body);
const sqlquery = `select hospital_id,count(psychiatrists.id) as psychiatrist_count,
sum(patients_count) as total_patients_count,hospital.name
from psychiatrists,hospital
where hospital.id=psychiatrists.hospital_id and hospital_id=${hospital_id};
select id,name,patients_count from psychiatrists
where hospital_id=${hospital_id};
`;
con.query(sqlquery, (err, result) => {
if (err) {
res.status(500).json("Some error while fetching");
console.log(err);
return;
}
console.log(result);
}
```
- handling disconnection due to idle timeout
```js
function handleDisconnect() {
con = mysql.createConnection(dbConfig); // Recreate the connection, since the old one cannot be reused.
con.connect(function onConnect(err) {
// The server is either down
if (err) {
// or restarting (takes a while sometimes).
console.log("error when connecting to db:", err);
setTimeout(handleDisconnect, 10000); // We introduce a delay before attempting to reconnect,
} // to avoid a hot loop, and to allow our node script to
}); // process asynchronous requests in the meantime.
// If you're also serving http, display a 503 error.
con.on("error", function onError(err) {
console.log("db error", err);
// if (err.code == "PROTOCOL_CONNECTION_LOST") {
// // Connection to the MySQL server is usually
// handleDisconnect(); // lost due to either server restart, or a
// } else {
// // connnection idle timeout (the wait_timeout
// throw err; // server variable configures this)
// }
handleDisconnect();
});
}
handleDisconnect();
con.on("connection", () => {
console.log("db connected");
});
```

52
joplin notion extension/Web Development/nodejs/ngrok.md Normal file
View file

@ -0,0 +1,52 @@
ngrok
https://reyhann.medium.com/how-to-expose-your-localhost-server-using-ngrok-9928ac2e26b8
https://www.npmjs.com/package/ngrok
- adding token through command
`ngrok config add-authtoken <TOKEN>`
```js
const PORT = process.env.PORT || 8080;
(async function () {
try {
await ngrok.authtoken(token);
const url = await ngrok.connect(PORT);
console.log("url: ", url);
} catch (err) {
console.log("err: ", err);
}
})();
```
- for multiple accounts in same machine
https://jd-bots.com/2022/03/04/how-to-use-multiple-ngrok-accounts-on-one-computer-in-windows/
`ngrok http 80 -config="C:\Users\jagdish\.ngrok2\my-ngrok-2.yml"`
format of `ngrok.yml`
```yml
version: "2"
authtoken: <AUTH_TOKEN>
```
- in shell
ngrok http 80 (here 80 is port number)
- in termux
to fix "failed get HOME directory"
first you need to type "whoami" without quote in your termux.
that will showed the user
~ $ whoami
u0_a185l
&nbsp;
copy the output (u0_a185l) for me.
than add the user to path by typing "export USER=u0_a185l"
~ $ export USER=u0_a185
enter.
and re-run ngrok
~ $ ./ngrok
or
https://github.com/Yisus7u7/termux-ngrok/
- using wifi instead of mobile data in termux
https://www.youtube.com/watch?v=EGaZ9SajayY
simply use proot-distro and use ngrok from there, ngrok if installed in termux will work just have to configure auth-token separately

18
joplin notion extension/Web Development/nodejs/npm modules.md Normal file
View file

@ -0,0 +1,18 @@
npm modules
- ytpl
to get information about youtube playlist
https://github.com/TimeForANinja/node-ytpl/
Exception : https://www.youtube.com/playlist?list=PLi01XoE8jYojRqM4qGBF1U90Ee1Ecb5tt
not working for this playlist.just returning title, channel id and count of videos, not the video URLs
- recommended versions
```json
"node-fetch": "^2.6.7"
```
- https://github.com/iqbal-rashed/ytdlp-nodejs : based on yt-dlp (not working)
https://www.npmjs.com/package/yt-dlp-wrap (working)
- okay so after further investigating, switching from version
```"ytdl-core": "github:khlevon/node-ytdl-core#v4.11.4-patch.2",```
to something like
```"ytdl-core": "npm:@distube/ytdl-core@^4.13.3",```
makes the "parse decipher" error go away and the files download properly

165
joplin notion extension/Web Development/nodejs/playwright.md Normal file
View file

@ -0,0 +1,165 @@
playwright
https://playwright.dev/docs/intro
- `npm i playwright` and `npx playwright install firefox` to download firefox
- Normal branded firefox wont work with playwright. playwright will download its own patch of firefox
&nbsp;
Simple example
```js
const { firefox } = require("playwright");
(async () => {
const browser = await firefox.launch({
// executablePath: "/usr/lib/firefox/firefox",
headless: false,
});
const page = await browser.newPage();
await page.goto("https://playwright.dev/");
// .DocSearch is a CSS selector for search button
await page.locator(".DocSearch").click();
// await browser.close();
})();
```
- In order to preserve profile use `launchPersistentContext` method instead of `launch`.
eg
```js
const { firefox } = require("playwright");
(async () => {
const browser = await firefox.launchPersistentContext(
"/home/hpes/.mozilla/firefox/r7e7y6qd.playwright",
{
// executablePath: "/usr/lib/firefox/firefox",
headless: false,
timeout: 0,
// args: ["--profile-directory=playwright"],
}
);
const page = await browser.newPage();
await page.goto("https://rumble.com");
// await page.locator(".DocSearch").click();
// await browser.close();
})();
```
- for uploading videos in rumble
```js
const promiseSetTimeout = (time) => {
return new Promise((resolve) => {
setTimeout(resolve, time);
});
};
const { firefox } = require("playwright");
const path = require("path");
const videoTitle = "The Witcher 3 evil soft first touches p3 toad price fight";
const tagsValue = "#gaming #witcher3";
const channelValue = "6448233";
const video_dirname =
"/mnt/hdd/windows/witcher 3 reocrded videos/1080p/original/missions/hos/evil soft first touches/";
const video_filename =
"The Witcher 3 evil soft first touches p3 toad price fight.mp4";
const thumbnail_dirname =
"/mnt/hdd/windows/witcher 3 reocrded videos/1080p/original/missions/hos/evil soft first touches/assets";
const thumbnail_filename =
"evil soft first touches p3 fighting toad prince.png";
(async () => {
const browser = await firefox.launchPersistentContext(
"/home/hpes/.mozilla/firefox/r7e7y6qd.playwright",
{
// executablePath: "/usr/lib/firefox/firefox",
headless: false,
timeout: 0,
// args: ["--profile-directory=playwright"],
}
);
// const page1 = await browser.newPage();
// const page2 = await browser.newPage();
// await page1.goto("https://rumble.com/upload.php");
// await page2.goto("https://rumble.com/upload.php");
// await page1.waitForSelector(".title");
// const videoCompleteh3Handle = await page1.locator(".title");
// // completion of upload successfully
// videoCompleteh3Handle.innerText() === "Video Upload Complete!";
const page = await browser.newPage();
await page.goto("https://rumble.com/upload.php");
// filling video title
await page.locator("#title").fill(videoTitle);
// filling primary category
await page
.locator(
".video-details > div:nth-child(7) > div:nth-child(1) > div:nth-child(1) > input:nth-child(2)"
)
.fill("Gaming");
// filling tags
await page.locator("#tags").fill(tagsValue);
const selectHandle = await page.locator("#channelId");
selectHandle.selectOption(channelValue);
const uploadVideoHandle = await page.locator("#Filedata");
uploadVideoHandle.setInputFiles(path.join(video_dirname, video_filename));
await promiseSetTimeout(3000);
// uploading custom thumbnail
const uploadThumbnailHandle = await page.locator("#customThumb");
uploadThumbnailHandle.setInputFiles(
path.join(thumbnail_dirname, thumbnail_filename)
);
await promiseSetTimeout(2000);
const uploadButtonHandle = await page.locator("#submitForm");
// go to next step for final submission
uploadButtonHandle.click();
await promiseSetTimeout(4000);
// checking checkboxes before final submission
// await page.locator("#crights").check();
// await page.locator("#cterms").check();
// await page.locator("#crights").click();
// await page.locator("#cterms").click();
// crights
await page
.locator("div.form-checkbox:nth-child(2) > label:nth-child(2)")
.click();
// cterms
await page
.locator("div.form-checkbox:nth-child(3) > label:nth-child(2)")
.click();
await promiseSetTimeout(2000);
// finally submitting it
// await page.locator("#submitForm2").click();
// await page.locator(".DocSearch").click();
// await browser.close();
})();
```

220
joplin notion extension/Web Development/nodejs/postgresql.md Normal file
View file

@ -0,0 +1,220 @@
postgresql
- Install pg module
https://node-postgres.com/
For multiple queries there r 2 methods
In Node.js, when using the `pg` library (node-postgres) to work with PostgreSQL databases, executing multiple queries consecutively is a common requirement. There are several ways to achieve this depending on the nature of your queries and how they should be executed (e.g., in parallel, in sequence, within a transaction). Below are some approaches:
#### 1. Sequential Execution (with async/await)
If your queries are dependent on each other and need to be executed in a sequential manner, you can use `async/await` syntax to execute them one after the other.
```javascript
const { Client } = require('pg');
const client = new Client({
// Your database connection parameters
});
const executeQueriesSequentially = async () => {
try {
await client.connect();
const result1 = await client.query('SELECT * FROM table1;');
console.log('Result 1:', result1.rows);
const result2 = await client.query('SELECT * FROM table2;');
console.log('Result 2:', result2.rows);
// Continue for more queries
} catch (err) {
console.error(err);
} finally {
await client.end();
}
};
executeQueriesSequentially();
```
#### 2. Parallel Execution
If your queries are independent of each other, you can execute them in parallel using `Promise.all`.
```javascript
const executeQueriesInParallel = async () => {
try {
await client.connect();
const promises = [
client.query('SELECT * FROM table1;'),
client.query('SELECT * FROM table2;'),
// Add more queries as needed
];
const results = await Promise.all(promises);
results.forEach((result, index) => {
console.log(`Result ${index + 1}:`, result.rows);
});
} catch (err) {
console.error(err);
} finally {
await client.end();
}
};
executeQueriesInParallel();
```
### 3. Using Transactions
For cases where your queries must be executed as a single atomic operation, you will want to use a transaction. This is particularly important i
&nbsp;
&nbsp;
## Defending from SQL injection
Also this way will prevent from situation like when we use `insert into tablename values ('${str}')` and if str is `i won't do` then it will like `insert into tablename values ('i won't do)` so strng will be `i won` as later is single quote closing in.
This method is called parameterized queries or prepared statements instead of manually constructing query strings which handles sql injection and escaping characters.
- For PostgreSQL (pg package example):
```js
const { Pool } = require('pg')
const pool = new Pool()
const text = 'INSERT INTO your_table(name) VALUES($1,$2)'
const values = [`there isn't anything to do`, `ok go`]
pool.query(text, values, (err, res) => {
if (err) {
console.log(err.stack)
} else {
console.log(res.rows[0])
}
})
```
- For MySQL (mysql package example):
```js
const mysql = require('mysql');
const connection = mysql.createConnection({
host: 'example.org',
user: 'your_user',
password: 'your_pass',
database: 'your_db'
});
connection.connect();
const post = {name: `there isn't anything to do`};
const query = connection.query('INSERT INTO your_table SET ?', post, (err, result) => {
if (err) throw err;
console.log(result);
});
connection.end();
```
&nbsp;
### Reconnecting database
Sometimes database gets disconnected. There r various ways like reconnecting ir but few things needed to be noted. When reconnecting sometimes we need to create new object too instead of just reconnecting
Eg:
```js
const pg = require("pg");
let pgClient
const createPGConnection=()=>{
pgClient=new pg.Client(conString);
pgClient.connect()
console.log('database connected')
}
const conString = process.env.CONNECTION_URI;
client.on("ready", async () => {
console.log(`Logged in as ${client.user.tag}!`);
});
createPGConnection()
pgClient.on('error',async(err)=>{
console.log('err: ',err)
await pgClient.end()
console.log('connection ended')
console.log('trying to reconnect')
await promiseSetTimeOut(5000)
await createPGConnection()
})
```
&nbsp;
**Pooling**
https://node-postgres.com/features/pooling
```js
import pg from "pg";
const { Pool } = pg;
const poolConfig = {
user: process.env.PGUSER,
password: process.env.PGPASSWORD,
database: process.env.PGDATABASE,
host: process.env.PGHOST,
port: process.env.PGPORT,
max: 20,
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 5000,
};
export const pool = new Pool(poolConfig);
pool.on("error", (error) => {
console.error("unexpected error on idle client:", error.message);
process.exit(1);
});
export const query = (text, params) => pool.query(text, params);
```
some testing
```js
const test = async () => {
try {
await query("SELECT 1");
console.log("database connected bro.");
return true;
} catch (err) {
}
};
```
&nbsp;
transactions
```js
import { Pool } from 'pg'
const pool = new Pool()
const client = await pool.connect()
try {
await client.query('BEGIN')
const queryText = 'INSERT INTO users(name) VALUES($1) RETURNING id'
const res = await client.query(queryText, ['brianc'])
const insertPhotoText = 'INSERT INTO photos(user_id, photo_url) VALUES ($1, $2)'
const insertPhotoValues = [res.rows[0].id, 's3.bucket.foo']
await client.query(insertPhotoText, insertPhotoValues)
await client.query('COMMIT')
} catch (e) {
await client.query('ROLLBACK')
throw e
} finally {
client.release()
}
```

204
joplin notion extension/Web Development/nodejs/snippets/json to sqlite db.md Normal file
View file

@ -0,0 +1,204 @@
json to sqlite db
```js
const jsonToSQL = (
json = "",
rootPath = ".",
dbfilename = "db.db",
tablename = "tablename",
primaryKeyField,
isAppendMode = false
) => {
const originalSchemaPath = `${rootPath}/originalSchema.json`;
const parsedSchemaPath = `${rootPath}/parsedSchema.json`;
const jsonData = JSON.parse(json);
const db = new Database(`${rootPath}/${dbfilename}`);
// console.log(typeof [1, 2]);
// console.log(jsonData);
// db.exec(`drop table if exists ${tablename}`);
// db.exec(`delete from ${tablename}`);
const isArray = Array.isArray(jsonData);
let finalData = jsonData;
// if not an array then converting to array
if (!isArray) {
finalData = Object.values(finalData);
}
let originalSchemaWrapper = {};
let parsedSchemaWrapper = {};
const originalSchema = {};
const parsedSchema = {};
// retrieving schema stored earlier
if (fs.existsSync(originalSchemaPath)) {
const value = JSON.parse(fs.readFileSync(originalSchemaPath));
originalSchemaWrapper = value;
Object.assign(originalSchema, value[tablename] ?? {});
}
if (fs.existsSync(parsedSchemaPath)) {
const value = JSON.parse(fs.readFileSync(parsedSchemaPath));
parsedSchemaWrapper = value;
Object.assign(parsedSchema, value[tablename] ?? {});
}
const createTable = () => {
if (finalData.length == 0) {
return;
}
const firstData = finalData[0];
if (typeof firstData != "object" || firstData === null) {
throw new Error("invalid format");
}
// key value pair of fieldname and type, both are strings
for (let key in firstData) {
originalSchema[key] = typeof firstData[key];
}
const sqlColumnsInfoArray = [];
for (let colname in originalSchema) {
// to check if current column name is of primary key or not
const isPrimaryKey =
primaryKeyField != undefined &&
primaryKeyField != null &&
primaryKeyField == colname;
// console.log(colname, " isPrimaryKey: ", isPrimaryKey);
if (originalSchema[colname] == "number") {
parsedSchema[colname] = "numeric";
} else if (originalSchema[colname] == "boolean") {
parsedSchema[colname] = "integer";
// everything else is text as text can even store numbers too after being parsed to string so useful for nulls
} else {
parsedSchema[colname] = "text";
}
sqlColumnsInfoArray.push(
`${colname} ${parsedSchema[colname]} ${
isPrimaryKey ? "primary key" : ""
}`
);
}
const sqlCreateQuery = `create table if not exists ${tablename} (
${sqlColumnsInfoArray.join(",")}
);`;
// console.log(sqlCreateQuery);
const result = db.exec(sqlCreateQuery);
};
const insertTable = (record) => {
if (typeof record != "object" || record == null) {
throw new Error("invalid format");
}
for (let key in record) {
if (!Object.hasOwn(originalSchema, key)) {
let newSqlColumnInfo = "";
originalSchema[key] = typeof record[key];
if (originalSchema[key] == "number") {
parsedSchema[key] = "number";
} else if (originalSchema[key] == "boolean") {
parsedSchema[key] = "integer";
// newSqlColumnsInfoArray.push(`${key} integer`);
// everything else is text as text can even store numbers too after being parsed to string so useful for nulls
} else {
parsedSchema[key] = "text";
// newSqlColumnsInfoArray.push(`${key} text `);
}
newSqlColumnInfo = `${key} ${parsedSchema[key]} `;
db.exec(`alter table ${tablename} add column ${newSqlColumnInfo}`);
}
}
const columns_string = Object.keys(record).join(",");
const valuesArray = [];
for (let column in record) {
// if text
if (parsedSchema[column] === "text") {
// check if value is null or not
if (record[column] === null) {
valuesArray.push(null);
}
// if value is string then simply pushing it
else if (typeof record[column] === "string") {
valuesArray.push(record[column]);
}
// if its number or array or object it will be stringified accordingly
else {
valuesArray.push(JSON.stringify(record[column]));
}
}
// if its boolean
else if (parsedSchema[column] === "integer") {
valuesArray.push(record[column] ? 1 : 0);
// if its number
} else {
valuesArray.push(record[column]);
}
}
// const valuesPlaceholderArray =
const values_placeholders_string = new Array(valuesArray.length)
.fill("?")
.join(",");
const insertStatement = db.prepare(
`insert into ${tablename} (${columns_string}) values (${values_placeholders_string}) `
);
const info = insertStatement.run(...valuesArray);
// console.log("info: ", info);
};
if (Object.keys(originalSchema).length === 0) {
createTable();
} else {
if (!isAppendMode) {
db.exec(`delete from ${tablename}`);
}
}
for (let record of finalData) {
insertTable(record);
}
// console.log("result: ", result);
// const insertStatement = db.exec(
// "create table if not exists channels (info text)"
// );
// console.log("insertStatement: ", insertStatement);
originalSchemaWrapper[tablename] = originalSchema;
parsedSchemaWrapper[tablename] = parsedSchema;
fs.writeFileSync(originalSchemaPath, JSON.stringify(originalSchemaWrapper));
fs.writeFileSync(parsedSchemaPath, JSON.stringify(parsedSchemaWrapper));
// console.log("done");
console.log(`converted json to table ${tablename} in db ${dbfilename}`);
};
```

267
joplin notion extension/Web Development/nodejs/snippets/padloc To Keepaas.md Normal file
View file

@ -0,0 +1,267 @@
padloc To Keepaas
`app.js`
```js
const fs =require('fs')
const {getRecords} = require('./helpers')
const filename= 'My_Vault_2024-05-13.csv'
// const filename= 'test.csv'
const outputDataFile = "output.xml"
let outputData=""
outputData+=`<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<KeePassFile>
<Meta/>
<Root>
<Group>
<Name>Root</Name>
<Notes/>
<IconID>48</IconID>
<Times/>
<IsExpanded>True</IsExpanded>
<DefaultAutoTypeSequence/>
<EnableAutoType>null</EnableAutoType>
<EnableSearching>null</EnableSearching>
<Group>
<Name>padloc</Name>
<Notes>containing prev padloc database</Notes>
<IconID>48</IconID>
<Times/>
<IsExpanded>True</IsExpanded>
<DefaultAutoTypeSequence/>
<EnableAutoType>null</EnableAutoType>
<EnableSearching>null</EnableSearching>
`
const records = getRecords(filename)
// function to map names of keys to match that of keepaas ones
const mapObjects=(obj)=>{
const newObj={}
for(let key of Object.keys(obj)){
if(obj[key].length===0){
continue;
}
// console.log('key to be mapped: ',key, ' value: ', obj[key])
switch(key){
case 'name': newObj['Title']=obj['name']; break;
case 'tags' : newObj['Tags']= obj['tags']; break;
case 'Username' : newObj['UserName'] = obj['Username']; break;
case 'Email Address' : newObj['Email'] = obj[key]; break;
case 'Plain Text' : newObj['Notes'] = obj[key]; break;
case 'Name' : newObj['UserName'] = obj[key]; break;
default : newObj[key] = obj[key]
}
}
// console.log('newObj: ',newObj)
return newObj;
}
// function to add escape sequqnce for reserved characters in xml
const mapValueForXML=(str)=>{
let newStr= str.replace(/"/g,'')
newStr = newStr.replace(/&/g,'&amp;')
newStr = newStr.replace(/</g,'&lt;')
newStr = newStr.replace(/>/g,'&gt;')
newStr = newStr.replace(/"/g,'&quot;')
newStr = newStr.replace(/'/g,'&apos;')
return newStr
}
for(let record of records){
const tags= record.tags.replace(/"/g,'')
let tagsXML=''
if(tags.length===0){
tagsXML='<Tags/>'
}
else{
tagsXML=`<Tags>${tags}</Tags>`
}
outputData+=` <Entry>
<IconID>0</IconID>
<ForegroundColor/>
<BackgroundColor/>
<OverrideURL/>
${tagsXML}
<Times/>
`
const mappedRecord = mapObjects(record)
// console.log('')
for(let key in mappedRecord){
// console.log('')
if(key==='Password'){
outputData+=` <String>
<Key>${key}</Key>
<Value ProtectInMemory="True">${ mapValueForXML(mappedRecord[key]) }</Value>
</String>
`
continue
}
outputData+=` <String>
<Key>${key}</Key>
<Value>${ mapValueForXML(mappedRecord[key]) }</Value>
</String>
`
}
outputData+=` <AutoType>
<Enabled>True</Enabled>
<DataTransferObfuscation>0</DataTransferObfuscation>
<DefaultSequence/>
</AutoType>
<History/>
</Entry>
`
}
outputData+=` </Group>
</Group>
</Root>
</KeePassFile>`
fs.writeFileSync(outputDataFile,outputData)
```
&nbsp;
`helpers.js`
```js
const fs =require('fs')
const getRecords=(filename)=>{
const data =fs.readFileSync(filename,{encoding : 'utf8'})
// normal, quotes
let state="normal"
const lines= data.split('\n')
// console.log('lines: ',lines)
// extracting headings and also removing last character ie \r from the line
let headings= lines[0].slice(0,lines[0].length-1).split(',')
console.log('headings: ',headings)
let valuei=0
let str=``
const parseLine=(line,obj)=>{
// parsing character by character
for(let i=0;i<line.length;i++){
// comma found in normal state
if(state==="normal"&& line[i]===','){
obj[headings[valuei]]=str
// resetting buffer for next value
str=``
// moving to next field index
valuei++
}
else if(state==="normal" && line[i]==='"'){
state="quotes"
str+=line[i]
}
else if(state==="quotes" && line[i]==='"' ){
state="normal"
str+=line[i]
}
else{
str+=line[i]
}
}
}
const records=[]
let obj={}
for(let i=1;i < lines.length;i++){
// const line=lines[i].slice(0,lines[i].length-1)
// removing `\r` from every record
const line=lines[i].replace('\r','')
parseLine(line,obj)
// if at end of line, state is normal then it means current record
// ends so store last field in object and reset values, pushing record in array
if(state==='normal'){
obj[headings[valuei]]=str
records.push(obj)
// resetting buffer for next value
str=``
// moving to next field index
valuei=0
obj={}
}
// if it in quotes then add newline character to current field
else{
str+='\n'
}
// const commaSeparateArray= lines[i].split(',')
// obj['name']= commaSeparateArray[0]
// obj['tags']= commaSeparateArray[1]
// obj['Username']=commaSeparateArray[2]
// obj['Password']=commaSeparateArray[3]
// obj['URL']=commaSeparateArray[4]
// obj['Email Address']=commaSeparateArray[]
// const nameRegex= /^(.*),.*/
// const tagsRegex = /^.*,("?.*"?),.*/
}
console.log('records: ',records)
return records
}
module.exports= {getRecords}
```

Some files were not shown because too many files have changed in this diff Show more